Automatic cancellation of worker pipelines when an child or infant pipeline is cancelled

[NJLangley]

When cancelling the parent or child pipelines (recursive cancel), the framework pipelines complete with a cancelled status, but the worker pipelines continue running (because cancelling does not allow cleanup). In the case where 50 workers in different data factories need cancelling too, this can be time consuming.

It would be nice if the framework had the option to define a cancellation policy, much like the continuation on error options, to allow cancellation of worker pipelines in the case an infant is cancelled.

I have been looking into pushing data factory events into event grid, to trigger a function for cancelled infants, that calls a new a cleanup pipeline to do just this. It cancels the worker, and waits for it to finish before updating the CurrentExecution table (only if required). I have PoC code that I can share.

[mrpaulandrew]

@NJLangley sounds interesting. Yes, would be interested to see it.

Or, what about if you request a cancellation by updating the CurrentExecution table yourself with a special status. 'RequestCancel' or something.

Then as part of the infant pipeline worker checking iterations (Until) if it picks up the request it does the cancellations for you using a function or whatever.

This way the parent stays in control of the cancellation request without being cancelled itself.

[NJLangley]

The 'RequestCancel' idea seems interesting, and definitely worth exploring some more.

Our main concern is that the BAU running of the system will be handed off to less technical users. They are likely to have access to control running of the framework pipelines, so they can run on-demand loads, or could cancel a load if there is a problem. We wanted to ensure if one of these users cancelled the parent or child pipelines, then the worker pipelines would not be left running in the background with them unaware.

This may be overkill for lots of implementations, but our implementation is built as an optional extra that can be ignored if not needed.

[mrpaulandrew]

85