search

tortoise / tortoise-orm

Familiar asyncio ORM for python, built with relations in mind
https://tortoise.github.io
Apache License 2.0
4.57k stars 378 forks source link

Question - how to configure Tortoise for transparent auto-reconnect? #1604

Open davidmcnabnz opened 4 months ago

davidmcnabnz commented 4 months ago

Is your feature request related to a problem? Please describe. PostgreSQL suffered a segfault of a worker process, causing my program using Tortoise to suffer exceptions when attempting to access data.

I couldn't find anything in the documentation about how to set Tortoise up to retry lost connections.

Describe the solution you'd like A set of options I can pass to Tortoise.init() to govern automatic reconnection behaviour.

Also, while reconnection is in progress, for any tasks with current transaction calls to accept keywords indicating whether they want an exception in event of connection entering "retry" state, or want to sleep until connection is re-established.

Describe alternatives you've considered Presently, my only straightforward option is that when a process notices that Tortoise can no longer talk to the back end database, to have the process self-terminate, so that when systemd re-launches it, it makes a fresh connection attempt with fresh state.

Additional context Any positive advice appreciated.

abondar commented 4 months ago

Tortoise does not have explicit support for connection retries, we had something like that long ago, but scraped it in favour of pooling used in db clients I am not sure what would be right course of action here, as we expect that pools sort that themselves

You can share your configuration and examples of errors, may be I will be able to look into it, but not sure if we will change something

davidmcnabnz commented 4 months ago

I'm now configuring Tortoise with a higher maximum pool size.

@abondar or anyone else, can you recommend best practice in cases where a Tortoise-based program suffers an unexpected shutdown of the underlying PostgreSQL database it's connected to?

What I'm currently doing is catching ConnectedRefusedError when attempting transactions. When this occurs, I'm rolling back partially completed processing, then terminating the process, to let systemd start it up again. On restart, my process sits in a loop repeatedly calling Tortoise.generate_schemas() then sleeping until this call succeeds.

Surely there's a better way. Keen for your suggestions.