dakotahorstman
commented
6 months ago I can confirm the server does put the flow run in a canceling state immediately upon the request from the UI on its end.
Repeatedly querying the server with the below code shows the above MRE enters the canceling state immediately upon being requested from the UI. However, this is not reflected in the run context, nor is the on_cancellation= hook called until the flow run is termed. If this is intended, then the documentation ought to be revised as it currently states: 'on_cancellation=' An optional list of callables to run when the flow enters a cancelling state.
import asyncio
import time
from pprint import pformat, pprint
from prefect import get_client
from prefect.server.schemas.filters import (
FlowRunFilter,
FlowRunFilterState,
FlowRunFilterStateType,
)
from prefect.server.schemas.states import StateType
async def do():
async with get_client() as client:
while not time.sleep(1):
flow_runs = await client.read_flow_runs(
flow_run_filter=FlowRunFilter(
state=FlowRunFilterState(
type=FlowRunFilterStateType(any_=[StateType.CANCELLING]),
),
)
)
pprint(pformat([vars(fr) for fr in flow_runs], indent=4, width=1), indent=4)
if __name__ == "__main__":
asyncio.run(do())To reproduce:
- Start a flow with the above MRE.
- Let it run for a few seconds, then spin up this state-check poll. You should see an empty list printed on the console every second.
- Cancel the MRE flow from the UI, and its info will be immediately and repeatedly printed to the poll console.
- The MRE will then be termed by prefect, and the console prints will return to an empty list.
- During this process observe the MRE's
print(context.flow_run.state)will continue to printRunning()until it is termed.
serinamarie
mthanded
CoffeeMark2
First check
Bug summary
Good day.
My team and I have spent the past week debugging a file corruption issue created by one of our Prefect 2.0 tasks that took down a production server. We've tracked this issue to an improperly closed file handle that occurred on our standard 'reboot' Sunday.
Part of the 'Reboot Sunday' procedure is to go through and request each of our few long-running flows be canceled from the UI. We've noted internally the, at best, dubious handling Prefect has of handling these cancellation requests, but we haven't had any issues until now.
What we observed is the following:
Crash detected! Execution was aborted by a termination signal.including the flow being the issue here as-is "normal". This left the file handle open when Prefect termed the process. This file is in a shared directory and caused a watchdog to term the worker entirely in an attempt to release the file after a period of time.to_deploymentandservefunctions. They are not workers like in pools.Prefect did not enter a 'Cancelling' state as indicated by the docs as the below output shows the flow run state remains at
Running()until it is termed. Theon_cancellation=hook is also not called until the flow is termed which is too late by that point.Reproduction
Error
Versions
Additional context
No response