Receive `ERROR: query timeout exceeded (240s)` errors intermittently when running a large amount of controls

[cbruno10]

When running steampipe check all in the GitHub Compliance mod, or steampipe check benchmark.cis_v150 in the AWS Compliance mod, we sometimes receive the ERROR: query timeout exceeded (240s) for controls.

For instance:

cbruno@M1P github % steampipe check all

GitHub Compliance ........................................................................................................................................................... 4,496 / 9,673 [==========]

+ CIS Software Supply Chain v1.0.0 .......................................................................................................................................... 4,496 / 9,579 [==========]

| + 1 Source Code ........................................................................................................................................................... 2,281 / 6,104 [=======   ]

| | + 1.3 Contribution Access ...............................................................................................................................................     4 /    18 [==        ]

| |   + 1.3.7 Ensure two administrators are set for each repository .........................................................................................................     1 /     1 [=         ]
| |   |
| |   | ERROR: query timeout exceeded (240s)

...

Summary

OK .................................................................................................................................................................................. 2,441 [===       ]
SKIP .................................................................................................................................................................................... 0 [          ]
INFO ................................................................................................................................................................................ 2,736 [===       ]
ALARM ............................................................................................................................................................................... 4,492 [=====     ]
ERROR ................................................................................................................................................................................... 4 [=         ]

TOTAL ....................................................................................................................................................................... 4,496 / 9,673 [==========]

The 4 errors were 4 controls receiving the same error across several benchmarks

[jwsapienza]

I also witnessed simlar timeouts with steampipe-mod-github-sherlock when running steampipe check all :

| + Pull request template should be added in each public repository ..............................................................................................................  1 /   1 [=         ]
| |
| | ERROR: query timeout exceeded (240s)
| |
| + Description should be set in each public repository ..........................................................................................................................  1 /   1 [=         ]
| |
| | ERROR: query timeout exceeded (240s)
| |
| + Website URL should be set in each public repository ..........................................................................................................................  1 /   1 [=         ]
| |
| | ERROR: query timeout exceeded (240s)
| |
| + Topics should be set in each public repository ...............................................................................................................................  1 /   1 [=         ]
| |
| | ERROR: query timeout exceeded (240s)
| |
| + Default branch should block force push in each public repository .............................................................................................................  1 /  38 [===       ]
| |
| | ERROR: timeout: context deadline exceeded
| |
| + Default branch should block deletion in each public repository ...............................................................................................................  1 /  39 [===       ]
| |
| | ERROR: timeout: context deadline exceeded
| |
| + Default branch protections should apply to administrators in each public repository ..........................................................................................  5 /  38 [===       ]
| |
| | ERROR: timeout: context deadline exceeded
| |
| + Default branch requires pull request reviews before merging in each public repository ........................................................................................  1 /  38 [===       ]
|
|   ERROR: timeout: context deadline exceeded
|
Summary

OK ..................................................................................................................................................................................... 29 [==        ]
SKIP .................................................................................................................................................................................... 0 [          ]
INFO .................................................................................................................................................................................. 122 [=======   ]
ALARM ................................................................................................................................................................................... 8 [=         ]
ERROR .................................................................................................................................................................................. 24 [==        ]

TOTAL ............................................................................................................................................................................ 32 / 183 [==========]

[idesofoctober]

@kaidaguerre do you have any insight or path to resolution or a workaround? I noticed that steampipe query timeout is off for all queries except benchmarks/controls (or something to that effect), is there a reason why that is the default? What are the implications of turning the query timeout off, do you know?

[e-gineer]

@idesofoctober Have you tried using the --query-timeout argument? That allows you to customize the timeout.

[idesofoctober]

@e-gineer I have been fiddling with that (found it yesterday)....disabled it, and I think there are some unintended consequences in that things are "running" without any clear indication of what is happening (I don't see any queries going into AWS, the log (default loglevel) doesn't have new entries, etc. I changed it to ten minutes for a test run today, that may have been helpful (ran into a different bug due to my script not handling a certain internal condition properly, so need to try again). Any suggestions? Is there a way to limit the number of connections/queries to AWS at once, looks like the log is indicating a max of 25, can I lower that somewhere, do you know?