nickgervasi / broken_record

MIT License
44 stars 14 forks source link

Reconnect to Database for Each Parallel Job Run #20

Closed danielh257 closed 5 years ago

danielh257 commented 5 years ago

Recently our HIValidations job has been failing with connection issues: https://jenkins.zp.int/view/Validations/job/HIValidation/2349/console

Screen Shot 2019-07-01 at 2 17 19 PM

Through some investigation, it looks like the Parallel gem has an interesting bug with Postgresql where it will disconnect from the database after each job run. http://ruby.zigzo.com/2012/01/29/the-parallel-gem-and-postgresql-oh-and-rails/

With the updates to how we send validation job errors to specific slack channels, we now will run multiple batches of jobs with the ExternalJobScheduler, which uses Parallel to schedule jobs. This means we will need to reconnect to the database before each job run in the ExternalJobScheduler too. (See the ParallelJobScheduler for an ex of having to reconnect to the Postgres database for each parallel job run)

Note: I bumped the ruby version but have not published the gem yet. Will publish once the PR is merged