'on_cancellation' Hook Not Triggered in Prefect 2.16.4

CoffeeMark2 commented 3 weeks ago

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Prefect documentation for this issue.
[X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

The on_cancellation hook in a Prefect flow is not being triggered when the flow is canceled. Although the workflow appears as canceled in the Prefect UI and client, the log message from the on_cancellation hook is not displayed, indicating that the hook is not being executed.

Reproduction

from prefect import flow, get_run_logger
from prefect.deployments import Deployment
import time

def hook_cancellantion(flow, flow_run, state):
    logger = get_run_logger()
    logger.info("here is hook cancellation")

@flow(
    name="cancellation_hook_test",
    on_cancellation=[hook_cancellantion],
)
def cancellation_hook_test():
    logger = get_run_logger()
    for i in range(10):
        logger.info(f"flow is running {i}")
        time.sleep(10)

deployment = Deployment.build_from_flow(
    flow=cancellation_hook_test,
    name="weekday",
    tags=['data-engineering'],
    work_queue_name="default",
)

deployment.apply()

After running the code above for a little while, I cancel the flow execution using the Prefect UI.

issue

The flow has been canceled. The expected log message "here is hook cancellation" from the on_cancellation hook should appear, but it does not.

Versions

Version:             2.16.4
API version:         0.8.4
Python version:      3.11.8
Git commit:          e3e7df9d
Built:               Thu, Mar 14, 2024 5:11 PM
OS/Arch:             win32/AMD64
Profile:             test
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.45.2

Additional context

I am using the Agent instead of the Worker to run the code deployment. Could this be the cause of the issue?

desertaxle commented 3 weeks ago

Thanks for the issue @CoffeeMark2! These hooks will fire more reliably in the more recent versions of perfect. Can you upgrade to the latest version and see if this behavior persists?

CoffeeMark2 commented 2 weeks ago

Thank you for your prompt response! Sadly, our company might take some time to complete the Prefect upgrade. Once we have updated to the latest version, I will get back to you with the results.

PrefectHQ / prefect