Open yevgenpapernyk opened 5 years ago
This is stems from the fact that SQLAlchemy is not thread safe and the db worker that comes in with Frontera generates batches not on the main thread (the one the frontier and all the backend models were initialized on)
A possible fix would be to just read from the message bus in a that other thread and then schedule the actual batch generation to be run on the main thread. This is what I do in a hacked version of the worker I use myself, but I am hesitant to release it now because the way I do it is quite inelegant and also add some custom logic for other things. Anyway, that's the general idea if you or anybody else are willing to hack.
I'm using PostgreSQL as backend for the workers.
The worker config is:
The requirements.txt is:
After a while I'm getting this error periodically: