xmatthias
commented
1 year ago Freqtrade doesn't handle it's sessions itself - but through SQLAlchemy's scoped_session() - which automatically handles sessions in a threat-aware manner.
Effectively, this means that we'll neither
- explicitly open X sessions
- explicitly close X sessions
but we'll rely on SQLAlchemy's logic and functionality to handle this for us. We do however make sure (especially when in the API server context) to roll back transactions api requests may have opened to avoid leaks like this.
If this is a problem / leaky behavior - it's most likely a bug in SQLAlchemy - at least i'd not be aware how / where we could be change something to improve behavior in this (other than manually handling sessions, which i'm not even going to start to look at, as it'd complicate the code beyond reason, and would probably introduce a lot more bugs in the process).
Looking at the logs / version - i'd also suggest to look at this with develop (and updated dependencies). SQLAlchemy was recently bumped from 1.4 to 2.0.x - which might improve behavior in this.
From the logs i also noticed that you're using freqAI - while not in itself a problem, you should make sure to limit it to N-1 parallelism (where N = your core count) - so 1 core is always available to process "random" requests.
Otherwise it's surely a possibility that web-requests remain "hanging" (you should see errors / timeouts in the UI) as freqtrade has no more resources available to actually handle the incoming requests.
if Postgres is on the same server (which it seems to be in your case) - the actually better value might be N-2 - to leave one core to postgres entirely. I'm not sure how postgres handles heavy resource contention ... whenever i ran Postgres in the past, it was the only heavy consumer on that system. If it can't get CPU - the 2 (freqtrade / postgres) might contend for the same resource (while freqAI is training happily on all cores) - causing sessions to fail closing (or sticking around longer than they should).
To be sure to exclude heavy load as problem, the ideal way would be to run a test without freqAI on that same system. Otherwise resource contention can't be excluded.
Smidelis
After some time and with some open trades there are still some sessions being open for multiple hours querying some old trade information (unfortunately i dont have detailed information about the query right now as i just started a fresh dry run, but i can provide those queries in case you are interested).
gaugau3000
Describe your environment
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.
Describe the problem:
Im using Freqtrade with PostgreSQL as database (
--db-url postgresql+psycopg2://db_user:*****@localhost:5432/Hasensachen_FreqAI_07). The database is running via docker. Freqtrade is running as systemd service. Both are running on the same host. Everything is running fine except, that after some time freqtrade crashes with the following error:raise exc.TimeoutError(sqlalchemy.exc.TimeoutError: QueuePool limit of size 20 overflow 40 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)When i started using postgresql i was using the default QueuePool of size 5 overflow 10. I have increased it to size 20 overflow 40 by changing the following line https://github.com/freqtrade/freqtrade/blob/a31045874ed51faf22f7672ec4b8823df59fe83a/freqtrade/persistence/models.py#L48 toengine = create_engine(db_url, pool_size=20, max_overflow=40, future=True, **kwargs)Unfortunately this does not prevent Freqtrade from crashing but postpones the crash. With the default QueuePool settings it was running fine for a few hours, with the increased sizes it's running around 1 - 2 days before it crashes.
Im running pgAdmin to manage my DB. From there I can clearly see that the database sessions are increasing over time (unfortunately there is no history available). Right now im just near my limit again (60 sessions). The following picture shows what happens after a restart of freqtrade.
After the restart the sessions are getting reduced to below 10 and then adding up to the limit again.
Below just as an example a few of the open sessions:
(Please excuse the screenshots)
After the restart they are gone. What i experienced is, that they are increasing in real-time when i click around in FreqUI.
Steps to reproduce:
Relevant code exceptions or logs
In the attached file
ft_db_error.logyou can see a full run until FT is crashing. ft_db_error.logRelated info
I've found #7364, which shows the same error but it were concluded to be a connection issue. Im pretty sure that there is no connection issue at my side as i can clearly see from the db sessions in pgAdmin that they are increasing over time and the bot crashes, as soon as they are reaching the QueuePool size limit. Also as they are running within the same host. Hardware is also no issue as at the time of the crash as there are plenty of free resources available to the host at the time of the crash.
Thanks a lot for your support!