Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.94k stars 441 forks source link

Health Checks - Timer trigger executed on unhealthy instance #10403

Open asierpn opened 2 months ago

asierpn commented 2 months ago

We have implemented Health Check in our Functions Apps deployed in a dedicated plan, if one of the instances is unhealthy, the functions host still redirects timer trigger functions to the unhealthy instance, while all calls to http trigger functions are correctly redirected to the healthy instances.

Repro steps

Enable Health Check and use a Timer Trigger and an Http Trigger function with 2 or more instances, forcing one of them to be unhealthy in the Health Check function implementation.

Expected behavior

The Timer Trigger function is redirected to one healthy instance.

Actual behavior

The Timer Tigger function sometimes is redirected to the unhealthy instance.

bhagyshricompany commented 2 months ago

Hi @asierpn thanks for reporting please share the all repro steps. and func app name,invocation id ,timestamp region where you deployed if.

asierpn commented 2 months ago

Here are the details:

Invocation Id: 367a988c-b426-4eb5-995b-00f9026b5119 Timestamp: 2024-08-16T08:50:00.0002035Z Region: North Europe

Repro steps:

Create two functions, one for the health check, which is hacked to return an unhealthy state on one of the two deployed instances:

    [Function("HealthCheck")]
    public async Task<IActionResult> HealthCheck([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "health")] HttpRequest req)
    {
      if (Environment.GetEnvironmentVariable("COMPUTERNAME") == "XXX")
      {
        return new ObjectResult(Enum.GetName(typeof(HealthStatus), HealthStatus.Unhealthy))
        {
          StatusCode = (int?)HttpStatusCode.ServiceUnavailable
        };
      }
      return new OkObjectResult(Enum.GetName(typeof(HealthStatus), HealthStatus.Healthy));
    }

And another one for the timer trigger:

    [Function($"TestFunction")]
    public void TestFunction([TimerTrigger("0 */1 * * * *")] TimerInfo timer)
    {
      Logger.LogInformation($"Test executed in {Environment.GetEnvironmentVariable("COMPUTERNAME")}");
    }

Deploy across at least two instances (we use Linux Containers), enable the health check in the Function App and check the logs, the TestFunction should only be redirected to the healthy instance.

asierpn commented 1 month ago

Is there any update on this issue at all?

bhagyshricompany commented 1 month ago

will checking update you soon.Thanks

bhagyshricompany commented 2 weeks ago

please share your log information details and invocation id,timestamp region etc.Thanks

asierpn commented 2 weeks ago

please share your log information details and invocation id,timestamp region etc.Thanks

I shared it on 23 August in this comment: https://github.com/Azure/azure-functions-host/issues/10403#issuecomment-2306340618 ....

bhagyshricompany commented 1 week ago

can you share which doc you refered?

asierpn commented 1 week ago

https://learn.microsoft.com/en-us/azure/azure-functions/configure-monitoring?tabs=v2#monitor-function-apps-using-health-check