We have a timer trigger which is supposed to run every 4 minutes.
For some reason, it failed to fire for a period of 32 minutes, and then resumed
Unfortunately for us, this timer trigger handles a critical batching process, and we cannot afford such pauses in operation.
Repro steps
Not sure this is easily reproducible, but I'll list the setup.
Provide the steps required to reproduce the problem
Create a timer trigger function that runs every 4 minutes.
Run the timer trigger function in a function at a scale of 10 instances.
Expected behavior
The timer trigger function should execute every 4 minutes, without gaps.
Actual behavior
The timer trigger function, at 2024-05-23T14:06:00Z in our instance, stops triggering on this instance and "pauses"
At 2024-05-23T14:38:00Z, the timer trigger starts triggering on another instance. Up to this point, the It seems that the trigger is moving between different instances, sometime around every 30 minutes. For some reason, this 30-odd minute gap appears where no instance [guessing] acquires the lease, so none executes it?
Our function app is a very complex one that uses change feeds from cosmos db in an event driven pattern. Items are processed through change feed events from input to output, ending up mainly as file blobs in blob storage containers.
Timer triggers provide a batching mechanism for output files to be grouped to the required size for downstream systems.
Can provide some detailed tracing information in DM's.
Because it appears that this problem cannot be duplicated with "vanilla" azure functions timers, but can with timers where durable azure functions extensions are used, we will close it here.
thank you.
Cross-posted from webjobs-extensions.
We have a timer trigger which is supposed to run every 4 minutes. For some reason, it failed to fire for a period of 32 minutes, and then resumed Unfortunately for us, this timer trigger handles a critical batching process, and we cannot afford such pauses in operation.
Repro steps
Not sure this is easily reproducible, but I'll list the setup.
Provide the steps required to reproduce the problem
Create a timer trigger function that runs every 4 minutes.
Run the timer trigger function in a function at a scale of 10 instances.
Expected behavior
The timer trigger function should execute every 4 minutes, without gaps.
Actual behavior
The timer trigger function, at 2024-05-23T14:06:00Z in our instance, stops triggering on this instance and "pauses" At 2024-05-23T14:38:00Z, the timer trigger starts triggering on another instance. Up to this point, the It seems that the trigger is moving between different instances, sometime around every 30 minutes. For some reason, this 30-odd minute gap appears where no instance [guessing] acquires the lease, so none executes it?
Known workarounds
None.
Related information
Our function app is a very complex one that uses change feeds from cosmos db in an event driven pattern. Items are processed through change feed events from input to output, ending up mainly as file blobs in blob storage containers.
Timer triggers provide a batching mechanism for output files to be grouped to the required size for downstream systems.
Can provide some detailed tracing information in DM's.