Open AkshayGoyal022 opened 4 years ago
cgratigny
commented
4 years ago @AkshayGoyal022 I ended up needing to follow the guidance here - https://fuzzyblog.io/blog/rails/2017/08/21/rails-apartment-tenancy-and-sidekiq.html
As the apartment-sidekiq gem hasn't been reliable for my setup. I have a database for each tenant, and as apartment isn't threadsafe, I had the same error you are experiencing.
krtschmr
commented
3 years ago @cgratigny can you please explain your thread safety issues?
i joined a codebase with heavy use of the apartment gem (almost everything in own schemas) and when sidekiq runs jobs, sometimes emails are sent from another clients domain and contain links to another clients url. this is obviously a threadsafe issue and my only solution is to destroy all software. i'm constantly pushing to revert the schema madness and do one shared database but i got pushed back a lot, so that ship has sailed i guess.
fernandomm
commented
3 years ago @krtschmr One "solution" is to run multiple sidekiq processes and use "-c 1" option.
You will loose the benefits of threads since each process will be using only one thread, but you should no longer experience these problems caused by Apartment + threads issues.
Depending on the type of job that you're running, the performance impact is low/none. In some cases you can achieve better results since you would be using multiple CPU cores ( https://medium.com/@zunaidahmed/sidekiq-and-scaling-dd6f1969b68e ).
krtschmr
commented
3 years ago Yes, it makes sense to have concurrency=1 to avoid a threading issue. we run 2 workers each 27 threads, so that 1 solution won't be possible right now. We also had class variables for the ClientSettings, so this might also cause an issue, i refactored this to instance variables now and hope no more wrong urls are included in emails. if it's still the case, i guess i need to destroy all software and restart.
We have 2 servers: master and a slave server. slave server is primarily being used to handle reports. All the long background running processes are pushed to the slave server. We are using sidekiq for background processing.
Now, whenever a report is ready we create a Download object in which the generated report is attached. As the process is running in slave we need to connect to master in order to create Download object. Code looks something like this:
But we are intermittently facing the error especially for long running jobs most probably because of sidekiq retrying that job.
MasterDbConnectionService code:
Config: Nginx, Phusion Passenger, Postgres, ruby-2.3.1, rails 5.1.3
Full stack trace: