thasso / pyjip

JIP Pipeline System
http://pyjip.readthedocs.org
Other
19 stars 8 forks source link

Having multiple pipeline instances at same time creates concurrency issues into the database #48

Open lfrias81 opened 10 years ago

lfrias81 commented 10 years ago

When running multiple pipelines at same time (like in the order of 50-100, each containing 10 jobs), the database encounters concurrency problems and leaves the jobs in running state, though they actually failed because problems with the database were found.

I consider the database should be able to cope with that level of concurrency, but otherwise, the state of the database should at least be keep coherent.