search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.57k stars 718 forks source link

Ability to cancel the chain of dependent tasks from within some task #1790

Open p-himik opened 6 years ago

p-himik commented 6 years ago

Let's assume there're tasks A, B, and C. And B and C depend on A. And all tasks depend on some parameters:

a: Delayed = A(params)
b: Delayed = B(a, params)
c: Delayed = C(a, params)

Any task may decide, given a particular input, that it doesn't want to continue computation. So, if A decides so, B and C won't even be started.

At this point, I think I can implement it by simply throwing an exception from the task that doesn't want to continue computation, and check for the exception when I check the future's result. The issue is that workers output all such exceptions.

Is it possible to implement such functionality in a manner that doesn't flood workers' output?

mrocklin commented 6 years ago

Currently no, but cancelling a task from within the task seems in-scope to me.

If you were intersted in implementing such functionality I would recommend looking at the Reschedule exception and how it is handled in the worker.py and scheduler.py files. It is inside the task scheduler, and so not particularly accessible, but this bit of the code is easier than most.

mrocklin commented 6 years ago

To be clear, I think it would be sensible for a task to raised a Cancel Exception and have it be handled in a special way.

p-himik commented 6 years ago

Thanks for the pointers, I'll take a look.

p-himik commented 6 years ago

Hmm, it appears to be much more involved than Reschedule because it cancels the graph. So it's like both Reschedule handling and an exception handling. And the latter appears to be handled just about everywhere. I think that implementing this feature would require me to understand the internals of distributed quite thoroughly.

mrocklin commented 6 years ago

You probably need Scheduler.stimulus_cancel, which should handle the graph manipulation on the scheduler side.

But no worries if you'd rather pass. This certainly isn't trivial.

On Mon, Feb 26, 2018 at 2:07 PM, Eugene Pakhomov notifications@github.com wrote:

Hmm, it appears to be much more involved than Reschedule because it cancels the graph. So it's like both Reschedule handling and an exception handling. And the latter appears to be handled just about everywhere. I think that implementing this feature would require me to understand the internals of distributed quite thoroughly.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1790#issuecomment-368611824, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszEl9A9P7z7hbZ0pOzj6nTq-7fOosks5tYwAQgaJpZM4STeqM .