Add a lock on tasks

[gwenzek]

I have a lot of long-running tasks that I'm running using doit.

I often accidentally launch a task while it was already running. Having two processes running the same task often lead to issues:

• use of the same resources
• second task will overwrite the results of the first task
• even if all goes well, I've wasted my CPU resources

I'd like to have a "locking" mechanism for a task. Is something like this already implemented ?

For now, I'm using a heuristic to check that a task isn't already running right at the beginning of my code. But it would be nicer if the runner could do this for me, by e.g. storing a lock in the DB when a task is launched. The lock would need some kind of deadline in case doit crash and is unable to remove it, and the lock should probably be ignored with the doit run -a flag.

Is there a hook to write something before a task is run ? AFAIU value_savers are only called after the task is run.

And of course, thanks for the nice tool!

[floer32]

@gwenzek I would suggest trying filelock if you haven't.

It is filesystem-based, which plays nicely with doit's handling of filesystem-based deps and targets. If your task does not currently involve files, you could simply touch files just to have this filelock-powered state. Also, you mentioned timeline/deadline. Well, filelock has a Timeout feature too :)

Some reasons this has an advantage over adding this to the doit database backends:

• It is a single implementation, rather than having to figure it out for the different DB backends.
• If anything goes wrong, <filename>.lock files can be inspected or deleted; it's a very simple model.
• It's concurrent safe, beyond doit. i.e. you can have your other Python processes respect filelock for some targets or deps.

---

As for a hook etc, you may want to subclass one of the actions, and put your hooks before/after execute's call to super(). That's one way to handle it anyways.

[floer32]

(Saying this in hopes that it helps in the short term... and hey maybe playing with the specific case, will create some sample code that could be made into a PR to doit ... 😸 )

[gwenzek]

Thanks for sharing this library.

I think this can be helpful for a workaround, but the main issue is I'm not sure what to do if I see a lock file.

• I don't want to mark the task as "uptodate" because I don't want to trigger depending tasks
• I don't want to return a fake "success" value because I don't want to trigger depending tasks
• I don't want to throw an error because it prevents doit to process other unrelated tasks

As a workaround I'm throwing an error if the lock file exists and use -c but I may end up skipping over other errors.

Currently a task has 3 possible status: {"up-to-date", "run", "error"} I'd like a 4th one: "delayed" which means that depending tasks can't be executed but that doit should continue processing other tasks.

Do you think that would be a reasonable design ?

[schettino72]

I guess for your use-case you better use a task-queue. celery or something similar... IMO doesnt make sense to try to handle this case internally on doit.

[gwenzek]

I guess for your use-case you better use a task-queue

Maybe my bug title isn't clear. But I'm already using a task queue and I'm able to tell if a task is already running. The issue I have is that even if I'm able to tell if the task is running, there is no way to pass this information to doit.

See my comment The dependency manager can only handle 3 states {"up-to-date", "run", "error"}. I'm proposing a 4th state "delayed" which means that the task is not ready to be run. Am I clearer ?

If you agree with this design I can send a PR.

[schettino72]

I'm proposing a 4th state "delayed" which means that the task is not ready to be run. Am I clearer ?

• you a know a task is running
• you ask doit to run it again
• but you don't want doit to execute the task

like that?i am assuming you have a complex dependency tree... otherwise you could just not run doit again. right?

not clear what would be the behavior of 'delayed'. doit already has a delayed meaning task will be executed at a later stage by same process, seems thats not what you want.

maybe what you are looking for is what doit calls 'ignore'... you would just need to to set it dynamicaly (not possible as of today)