Previously all the jobs where run in series and thus greatly slowed down the batches. Instead each job will now be ran in a separate thread. This means both the ssh connection and db session is now concurrent.
Care needed to be taken to ensure all db objects and sessions are created/used/closed within the same thread. Sharing objects between threads is not possible unless they are correctly merged across. Also scoped_session are required to ensure each thread has its own db session.
Currently 10 jobs can be ran in parallel. Fixes #45
Previously all the jobs where run in series and thus greatly slowed down the batches. Instead each job will now be ran in a separate thread. This means both the
sshconnection anddbsession is now concurrent.Care needed to be taken to ensure all
dbobjects and sessions are created/used/closed within the same thread. Sharing objects between threads is not possible unless they are correctly merged across. Alsoscoped_sessionare required to ensure each thread has its own db session.Currently 10 jobs can be ran in parallel. Fixes #45