Timer-Triggered Functions Suddenly Stopped Triggering

[BrunoJuchli]

Investigative information

• Timestamp: 2022-09-27T13:55:45.1227038Z
• Function App version: 4.10.4.4
• Function App name: sm-3rdparty-import-functions-test
• Function name(s) (as appropriate):
  • Zaptec-StartImportPartition0 ... Zaptec-StartImportPartition59 (again, 60 of those, too)
  • Easee-StartImportPartition0 ... Easee-StartImportPartition59 (there's 60 of those)
• Invocation ID:
  • 5d94aa5a-c8b8-44cd-bb9e-354456085b27 with timestamp: 2022-09-27 13:53:45.196 Zaptec-AllMetersOfSingleSmartMeUserImportFunction function which is executed via one of the timer triggered functions.
  • bbe48de37d0f42f091daf964e4e3d0cb with completed timestamp 2022-09-27T13:55:45.1227038Z ExecutionID (as found in the *Instances table of the storage the functions uses) of last running function Zaptec-PartitionImportOrchestrationFunction started via Zaptec-StartImportPartition59
• Region: Switzerland-North

Repro steps

This has happened out of of the blue after running the app for about 31 hours since last deployment. Before, the app ran without issue for months (deployments were dependency updates only).

Therefore, I'm not sure what the steps to reproduce are. I'll give an overview of the application instead:

There's a total of 120 timer triggered functions:

Zaptec

The Zaptec-StartImportPartition* functions are intended to be run every 15 minutes each. There's 60 functions which have a 15 seconds offset from each other. That is, every 15 seconds one of the Zaptec* functions should be run, starting with Zaptec-StartImportPartition0, then Zaptec-StartImportPartition1 after 15 seconds and so on.

All these functions do is using IDurableOrchestrationClient to StartNewAsync another function called Zaptec-PartitionImportOrchestrationFunction passing in the partition number (The * in the name of the timer triggered `Zaptec-StartImportPartition* function). This function then does the actual work.

Easee

The Easee-StartImportPartition* functions work similarly to the Zaptec once with the difference that there are 60 functions run once every hour with a 1 minute offset from each other.

Expected behavior

The timer triggered functions should continue to be executed periodically.

Actual behavior

All timer triggered functions of the function app have stopped executing after 2022-09-27T13:55:45.1227038Z Function execution count:

Known workarounds

Possibly restarting the function app. (I didn't to it because I thought someone might want to investigate first. This happened on our test system so we do have time to investigate further).

Related information

• Programming Language: C#
• Package References:
  • Microsoft.NET.Sdk.Functions 4.1.3
  • Microsoft.Azure.WebJobs.Extensions.DurableTask 2.8.1
  • Microsoft.Azure.WebJobs.Extensions.Storage 5.0.1
  • Microsoft.ApplicationInsights.WorkerService 2.21.0
  • ... and more

[ramya894]

@BrunoJuchli We are investigating this issue and let you know the findings soon.

[kashimiz]

Bringing this up with the product group for discussion/investigation; will update.

[BrunoJuchli]

Note that on September 29th between 1:15 and 1:45 UTC the timer triggered functions have re-started to execute without any action from our side. (we have automated deployments triggered by Dependabot, but this has only run on October 3rd and not on September 29th).