Azure / azure-webjobs-sdk

Azure WebJobs SDK
MIT License
739 stars 358 forks source link

CRITICAL Azure function timer trigger operating every 4 minutes "pauses" for around 32 minutes, then resumes operating #3076

Closed ericleigh007 closed 5 months ago

ericleigh007 commented 5 months ago

Cross-posted from webjobs-extensions.

We have a timer trigger which is supposed to run every 4 minutes. For some reason, it failed to fire for a period of 32 minutes, and then resumed Unfortunately for us, this timer trigger handles a critical batching process, and we cannot afford such pauses in operation.

Repro steps

Not sure this is easily reproducible, but I'll list the setup.

Provide the steps required to reproduce the problem

  1. Create a timer trigger function that runs every 4 minutes.

  2. Run the timer trigger function in a function at a scale of 10 instances.

Expected behavior

The timer trigger function should execute every 4 minutes, without gaps.

Actual behavior

The timer trigger function, at 2024-05-23T14:06:00Z in our instance, stops triggering on this instance and "pauses" At 2024-05-23T14:38:00Z, the timer trigger starts triggering on another instance. Up to this point, the It seems that the trigger is moving between different instances, sometime around every 30 minutes. For some reason, this 30-odd minute gap appears where no instance [guessing] acquires the lease, so none executes it?

timer-pause-nft

Known workarounds

None.

Related information

Our function app is a very complex one that uses change feeds from cosmos db in an event driven pattern. Items are processed through change feed events from input to output, ending up mainly as file blobs in blob storage containers.

Timer triggers provide a batching mechanism for output files to be grouped to the required size for downstream systems.

Can provide some detailed tracing information in DM's.

ericleigh007 commented 5 months ago

Because it appears that this problem cannot be duplicated with "vanilla" azure functions timers, but can with timers where durable azure functions extensions are used, we will close it here. thank you.