PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.66k stars 1.65k forks source link

on_cancellation hook not run due to `Process was terminated due to a Ctrl+C or Ctrl+Break signal` #15912

Open tnguyen921 opened 1 month ago

tnguyen921 commented 1 month ago

Bug summary

  1. When cancelling a flow run, the on_cancellation handler is not run.
  2. reproducible example. This was given to me by Bianica on Slack, and it runs on her machine
    
    from prefect import flow
    from prefect.logging.loggers import flow_run_logger
    import time

def log_something(flow, flow_run, state): flow_run_logger(flow_run, flow).info(f"Hey there Tri, flow {flow.name} cancelled")

@flow(on_cancellation=[log_something], log_prints=True) def test_flow(): time.sleep(300) # leave ample to cancel return 42

3. Stack trace. Note the `^C` at the beginning of line 2. I don't put it there. It always happen like that whenever I cancel

22:28:07.264 | INFO | prefect.worker.process.tnguyen-1 - Found 1 flow runs awaiting cancellation. ^C22:28:07.656 | ERROR | prefect.flow_runs.worker - Process 28292 exited with status code: 3221225786; Process was terminated due to a Ctrl+C or Ctrl+Break signal. Typically, this is caused by manual cancellation. 22:28:07.659 | INFO | prefect.flow_runs.worker - Cancelled flow run 'e088b6bb-36b6-4591-8752-6068ff2bb93e'!

4. I tested on both Prefect self-hosted and Prefect cloud. I run the worker as a local process on powershell. 

**_I've tested the cancellation ~ 50 times. And only once have I seen it correctly call the `on_cancellation` hook correctly._**

### Version info

```Text
Version:             3.0.4
API version:         0.8.4
Python version:      3.13.0
Git commit:          c068d7e2
Built:               Tue, Oct 1, 2024 11:54 AM
OS/Arch:             win32/AMD64
Profile:             cloud
Server type:         cloud
Pydantic version:    2.9.2

Additional context

No response

shuzhiwei commented 3 days ago

I had the same problem.