[ASE-146] Fix civis parallel backend bug - Githubissues

civisanalytics / civis-python

Civis API Python Client

BSD 3-Clause "New" or "Revised" License

34 stars 26 forks source link

[ASE-146] Fix civis parallel backend bug #395

Closed evanderh closed 4 years ago

evanderh commented 4 years ago

When using civis.parallel.make_backend_factory with joblib, cancelling a child job fails to cancel all the remaining child jobs and hangs indefinitely

To reproduce:

Check out the minimal example: git checkout 38b0e02
Run example reproduce script: ./examples/parallel_cancel.py
Through the platform UI, cancel the first job

What's going wrong:

The first job gets cancelled, calling _CivisBackend.abort_everything which then should cancel all the other jobs
The second job is cancelled through the api in ContainerFuture.cancel
In the second job's callback, future.cancelled() == False, because the future hasn't resolved yet (I think?) and isn't aware that is_cancel_requested == True
This causes the third job to not get cancelled and the parent script hangs open indefinitely

Fix:

Disclaimer: This likely isn't the best fix but I just wanted to try an initial fix to see if it worked
I made the callback aware of is_cancel_requested, which correctly causes the the third job to get cancelled
The parent script correctly raises a CancelledError instead of hanging indefinitely

mheilman commented 4 years ago

I think the CI errors might be resolved by bringing the branch up-to-date with master. See #394