Open dazza-codes opened 3 years ago
This might be solved by limiting configs to using a single client in any connection pool, so that clients cannot become stale in the pool. e.g.
aio_batch_config = AWSBatchConfig(
aio_batch_db=jobs_db,
min_pause=20,
max_pause=40,
start_pause=60,
max_pool_connections=1,
sem=500,
)
asyncio.run(aio_batch_monitor_jobs(jobs=jobs, config=aio_batch_config))
This is related to https://github.com/aio-libs/aiobotocore/issues/864
A batch job monitor can crash hard due to an expired signature in the client, e.g.
It might be possible to work around this by adding options to increase a connection or read timeout, e.g.
Another similar pattern uses a default config:
The monitoring code might need to detect and catch exceptions for invalid signatures. It could replace the client with a new one, or find some way to update the signature for a client.