Background tasks are again blocking

[ghukill]

After v0.3 updates for rerouting some background tasks behavior, they are again blocking.

[ghukill]

Related, possibily never fully addressed: https://github.com/WSULib/combine/issues/200

[ghukill]

Rooting back through commits, traced it back to when ensure_active_session_id() was introduced: https://github.com/WSULib/combine/commit/9df081646a1e32da726e964fd4f31a0db0351698#diff-ac449b072cfbaa2fadb1dedc098de4cdR295

[ghukill]

Improved, but not solved. Problem of background tasks blocking runs deep and might require some relatively major changes.

The crux of the problem is that any .save() to Job instance will lock the DB where that table is queried. This can be mitigated by putting numerous, quick running operations at the end of task, after the long-running spark task, and it's borderline unoticeable. But, it's there. This happened in this instance where more functionality was added, pre-spark task, and the locking was obvious.

A couple of approaches and factors come to mind:

• need better bg task system, would have opportunity to revisit DB connections with Celery
• move Job table -- all tables? -- to Mongo, which would not present this problem

[ghukill]

Will be addressed by switching to Celery: https://github.com/WSULib/combine/issues/184

[ghukill]

Indeed, Celery for bg tasks fixes.

At the cost of introducing some race conditions and different threads/versions of model instances. But this is a problem that can be mitigated with attention to refresh_from_db(). The blocking made Combien entiurely unusable, this will require more attention to hooks and refreshing.

Closing.