search

dchackett / taxi

Lightweight portable workflow management system for MCMC applications
MIT License
3 stars 1 forks source link

MIA taxis and abandoned tasks #13

Closed dchackett closed 6 years ago

dchackett commented 7 years ago

When a taxi dies unexpectedly, it leaves an inconvenient mess behind. The task it was working on remains marked "active", but is actually abandoned. This requires the user to fix the task by hand.

Solution: the "taxis" table contains a registry of all currently-running taxis. Occasionally (between each task), each taxi checks the queue to make sure all registered taxis are present. If any are missing, un-abandon their task and mark them as MIA in the taxi registry (to call user attention to the issue).

For anti-thrashing purposes, each task should probably have some sort of "n_times_abandoned" counter. If it goes above some threshold, e.g. 2 or 3 attempts, the taxis will mark it "failed" or "unrunnable" instead of un-abandoning it to be tried again.

dchackett commented 6 years ago

taxi now detects when a taxi is missing, marks that taxi missing ('M'), and marks its task abandoned ('abandoned').

Leaving issue open; still need to play with automatic/adaptive handling of this situation.