Health check UI is not showing nor recording status history when using SQL server storage provider

[AndrzejKroczynski]

What happened: Health check UI is not showing nor recording status history when using SQL server storage provider.

What you expected to happen: Health check UI shows and records history in the same way as when using in memory storage provider (which works just fine)

How to reproduce it (as minimally and precisely as possible):

• create Web API project
• add health check UI as per code below
• start the project
• trigger status change on SampleCheck endpoint
• check if history is displayed, check if HealthCheckExecutionHistories is populated

Source code sample: Startup.cs of .Net core API

 services
                .AddHealthChecksUI(setupSettings: setup =>
                {
                    setup.AddHealthCheckEndpoint("SampleCheck", "localhost:5111/status/ui");                    
                    setup.MaximumHistoryEntriesPerEndpoint(500);
                    setup.SetEvaluationTimeInSeconds(30);
                })
                .AddSqlServerStorage(Configuration.GetConnectionString("HealthChecksUIContext"));

Connection to db works fine, tables are created, other data is stored correctly, but HealthCheckExecutionHistories table remains empty.

Anything else we need to know?:

Environment:

• .NET Core version: 3.1
• Healthchecks version: 3.1.1
• Operative system: Widows 10
• Others:

[CarlosLanderas]

Hello @AndrzejKroczynski, could you show your localhost:5111/status/ui endpoint healthchecks configuration?.

Thanks!

[AndrzejKroczynski]

Hi @CarlosLanderas,

The startup code is:

//ConfigureServices
services.AddStatusChecks()
    .AddSqlServer(connectionString, name: "sql", failureStatus: HealthStatus.Unhealthy, tags: new[] { "detail" });
//there are more similar checks added

//Configure
app.UseHealthChecks("/status/ui", new HealthCheckOptions
            {
                ResultStatusCodes =
                {
                    [HealthStatus.Healthy] = StatusCodes.Status200OK,
                    [HealthStatus.Degraded] = StatusCodes.Status500InternalServerError,
                    [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable
                },
                Predicate = (check) => check.Tags.Contains("detail"),
                ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse,
                AllowCachingResponses = false
            });

Also, see below sample output if you would like to mock it: {"status":"Healthy","totalDuration":"00:00:01.8635091","entries":{"sql":{"data":{},"duration":"00:00:00.0060081","status":"Healthy"},"anotherDb":{"data":{},"duration":"00:00:00.0036762","status":"Healthy"},"url1":{"data":{},"duration":"00:00:00.0178600","status":"Healthy"},"url2":{"data":{},"duration":"00:00:00.0190634","status":"Healthy"},"url3:{"data":{},"duration":"00:00:00.0232293","status":"Healthy"},"url4":{"data":{},"duration":"00:00:00.0218745","status":"Healthy"},"url5":{"data":{},"duration":"00:00:00.0277542","status":"Healthy"},"serviceBus_Q1":{"data":{},"duration":"00:00:00.3649055","status":"Healthy"},"serviceBus_Q2":{"data":{},"duration":"00:00:01.3788014","status":"Healthy"}}}

Also, note that localhost:5111 is on .Net Core 2.2 (cannot upgrade it yet due to other dependencies), so it also uses Health Check UI in 2.2.* version. Could that be the problem? Other than issues with SQL Server storage this mix seems to work fine...

[CarlosLanderas]

@AndrzejKroczynski the status history timeline only writes a new entry when the state switches from healthy to unhealthy and viceversa (so it is not writing continuously when the state is the current one). Are you aware of that?.

Are you trying this forcing the sql to be healthty-unhealthy-healthy?

As you can see here:

[AndrzejKroczynski]

@CarlosLanderas - yep - I am aware about that.

This works perfectly fine when I'm using in memory storage provider (this looks exactly as on screen you've provided).

The problem is than when I switch to SQL server storage provider it stops working so I only see the current status, but no time line showing how the status was changing (even though it actually changed coupe of times). There are also no records in HealthCheckExecutionHistories table (I guess this is where those changes should be kept).

So the summary is that if I use AddInMemoryStorage() all works fine, but with AddSqlServerStorage(...) history does not work (all other code is exactly the same).

[CarlosLanderas]

Thanks for the info! I'll take a look into it. It's really weird as the code running is the same for all providers

[CarlosLanderas]

@AndrzejKroczynski I am not able to reproduce the issue. I am using the StorageProviders sample:

https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/tree/master/samples/HealthChecks.UI.StorageProviders

and everything is working fine:

One question. Are you using the same sql instance for the SQL HealthCheck project and the UI project?. Maybe you are shutting down sql server for the healthcheck and the UI is not able to write as well?. That would explain why in memory is working

[AndrzejKroczynski]

@CarlosLanderas - I'm using separate instance for Health Check UI.

Let me checkout the sample as well and see what is the difference between mine code and the sample.

[CarlosLanderas]

That sample runs local Healthchecks and UI. I am going to try to reproduce this using different processes

[AndrzejKroczynski]

@CarlosLanderas - checked the sample with local UI checks and it works perfectly fine, but when I switched to remote service (e.g. localhost:5111) it stopped working.... The JSON returned by both local and remote api seems to be in the same format...

[rajeshkandati]

Issue: Health check UI is not showing TimeLine and HealthCheckExecutionHistories table is empty.

Steps followed: 1) Created asp.net core 3.1 API and added AddHealthChecks under ConfigureServices as per below. public static class HealthCheckExtensions { ///

/// Add Health Checks ///

/// /// /// public static void AddHealthChecksService(this IServiceCollection services, IConfiguration configuration, IWebHostEnvironment env) { services.AddHealthChecks() .AddCheck(env.ApplicationName.Trim()); }

    /// <summary>
    /// Random Api check
    /// </summary>
    private class RandomHealthCheck : IHealthCheck
    {
        public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
        {
            //if (DateTime.UtcNow.Minute % 2 == 0)
            {
                return Task.FromResult(HealthCheckResult.Healthy());
            }

            //return Task.FromResult(HealthCheckResult.Unhealthy(description: $"The healthcheck {context.Registration.Name} failed at minute {DateTime.UtcNow.Minute}"));
        }
    }
}

2) Updated UseEndPoints as per below.

app.UseEndpoints(endpoints => { endpoints.MapControllers(); endpoints.MapHealthChecks("/healthz", new HealthCheckOptions { Predicate = r => r.Name.Equals(env.ApplicationName.Trim(), StringComparison.InvariantCultureIgnoreCase), ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse }); });

3) Created One more API and updated appsettings.json as per below "HealthChecksUI": { "HealthChecks": [ { "Name": "API1", "Uri": "https://API1.azurewebsites.net/healthz" } ], "Webhooks": [
], "EvaluationTimeInSeconds": 10, "MinimumSecondsBetweenFailureNotifications": 60, //"HealthCheckDatabaseConnectionString": "Data Source=[PUT-MY-PATH-HERE]\healthchecksdb",
}

4) Updated ConfigureServices(API 2) as per below services.AddHealthChecks(); //adding health check UI services services.AddHealthChecksUI(setup => { // Set the maximum history entries by endpoint that will be served by the UI api middleware //setup.MaximumHistoryEntriesPerEndpoint(50); }).AddSqlServerStorage(sConnection);

5) Updated UseEndPoints under Configure(API 2) as per below app.UseEndpoints(config => { config.MapHealthChecks("/healthz", new HealthCheckOptions { Predicate = _ => true, ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse });

            config.MapHealthChecksUI(setup =>
            {
                setup.UIPath = "/healthchecks-ui"; // this is ui path in your browser
                setup.ApiPath = "/health-ui-api"; // the UI ( spa app )  use this path to get information from the store ( this is NOT the healthz path, is internal ui api )
                setup.AddCustomStylesheet("wwwroot/healthcheck.css");
            });

            config.MapDefaultControllerRoute();
        });

Let me know if i am missing anything to persist executionhistories and to show Timeline.

[CarlosLanderas]

@AndrzejKroczynski @rajeshkandati

I've created a sample from scratch using nuget packages and two different sql instances. You can find the source code sample here:

https://github.com/CarlosLanderas/healthchecks-sample

(There is a docker-compose file to easily start the two sql instances)

Everything is working fine for me. Could you give it a try and compare with your codebase?.

In the docker compose folder execute:

docker-compose stop sqlserver2
docker-compose start sqlserver2

two switch health status.

Thanks!

[AndrzejKroczynski]

@CarlosLanderas - Your sample with two sql instances works just fine, but I think it helped me to identify the problem.

I think the issue is that status history is only saved when OVERALL component status changes. For example if we have health check end point with 10 checks where one of the checks constantly fails, then overall status is constantly Unhealthy - but other 9 checks are still performed, but their history is not saved.

For example, if you replace sql check with:

  services
      .AddRouting()
      .AddHealthChecks()
      .AddCheck(name: "random",
              () =>
              {
                   return DateTime.UtcNow.Second % 2 == 0
                       ? HealthCheckResult.Healthy()
                        : HealthCheckResult.Unhealthy(description: "Test description here");
                })
       .AddCheck(name: "constant",
               () =>
               {
                   return 
                       HealthCheckResult.Unhealthy(description: "Test description here");
        });

In that case "random" check changes are not being tracked, because overall status remains unchanged.

[rajeshkandati]

@CarlosLanderas - Same here. Sample is working fine for me too. Is it possible to find if the HealthCheckEndpoint is available or not(Alive or dead) from the HealthCheckSample?

[AndrzejKroczynski]

@CarlosLanderas - were you able to reproduce this on your side?

[CarlosLanderas]

Hello @AndrzejKroczynski and @rajeshkandati . Unfortunately I'm having very busy days. I'll try to check this tonight :). Thanks!

[woeterman94]

Can confirm that I have the same issue when using a postgress database as storage.

[xico002]

yup same problem with postgresql. Sometimes it saves history but other times it doesnt.

[xico002]

Anyone found any solution to this problem?

[CarlosLanderas]

@AndrzejKroczynski exactly, I need to check the code because this part was done by @unaizorrilla but I think the intention of the status history is recording the Overall endpoint health status, and not at the individual level.

[AndrzejKroczynski]

@CarlosLanderas - thanks for answer.

The history icon is displayed next to each individual item, so it suggests that history tracking is per each individual item.

Anyways - from usability perspective it would be really nice if we track that on individual item level.

[CarlosLanderas]

Then we have two options. Moving the details icon to the endpoint header or saving history at the individual level.

@unaizorrilla what do you think?

[AndrzejKroczynski]

@CarlosLanderas @unaizorrilla - I would really appreciate if you decide to record that on individual level, because it adds usability in real-life scenario.

For example - I have web service with 10 health checks - one of health checks is constantly failing because third-party service has issue. But the service that is failing is not critical, so I can live with that. In that case if another health check - this time critical is failing every now end then history will not show that because overall status doe not change.

[LeonT27]

Hello,

I'm having the same problem with history, when I have multiple checks configure and one is constantly changing to healthy and unhealthy, the history of that element is not present but, when I only leave one check in the configuration, the history is present.

Like @AndrzejKroczynski said, I think is better for the user to record individual history of checks.

[netcorenewbee]

Hello, I have a similar issue when using SQLServer Storage. This table dbo.HealthCheckExecutionHistories contains history of only one endpoint out of 11 I have configured.

If the status changes for that one particular endpoint it is logged correctly, but the rest is omitted every single time.

[galaldev]

I have the same problem

[balayoglu]

Hi all,

Seems the problem hasn't been solved yet. I tried both InMemoryStorage and SqlServerStorage, but the UI doesn't display the historical changes on the statuses.

These are the package versions I have used:

    <PackageReference Include="AspNetCore.HealthChecks.SqlServer" Version="5.0.2" />
    <PackageReference Include="AspNetCore.HealthChecks.UI" Version="5.0.1" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.Client" Version="5.0.1" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.InMemory.Storage" Version="5.0.1" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.SqlServer.Storage" Version="5.0.1" />

[rahul230691]

Does recording status history timeline not work with following packages? It shows me with "Healthy" or "Unhealthy". No history graph.

<ItemGroup>
    <PackageReference Include="AspNetCore.HealthChecks.AzureServiceBus" Version="5.1.1" />
    <PackageReference Include="AspNetCore.HealthChecks.Redis" Version="5.0.2" />
    <PackageReference Include="AspNetCore.HealthChecks.UI" Version="3.1.3" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.Client" Version="3.1.2" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.InMemory.Storage" Version="3.1.2" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.SqlServer.Storage" Version="3.1.2" />
    <PackageReference Include="AspNetCore.HealthChecks.Uris" Version="5.0.1" />
  </ItemGroup>

[s4lvo]

Hi all, has anyone found a solution? Seems the problem hasn't been solved yet

[OneCrazyRussian]

I am having a similar issue but i've found out that the way I've tested for negatives also broke my endpoint entirely: HealthChecks.UI.Core.HostedService.HealthCheckReportCollector: Error: GetHealthReport threw an exception when trying to get report from /health configured with name Health Checks API. So it went like this: first no history until the error happened, then UI starts showing error (still no history) then when /health endpoint comes back to life UI shows that it's fine and again - no history. Which is kinda defeating the purpose of the whole monitoring thing if failed monitoring does not count as an error and is not written in history

[marcasmar94]

Any updates on this ?

[nathan-hitchman]

Do we have an ETA on this?

[sungam3r]

Nope. See #1714

[jsanjuan2016]

Same problem here. I am using SQLite...

[jsanjuan2016]

As workaround I ended up creating a trigger on the database (in my case Sqlite):

CREATE TRIGGER after_update_executions 
   AFTER UPDATE ON Executions
   WHEN old.Status <> new.Status
BEGIN
    INSERT INTO HealthCheckExecutionHistories ([Name], [Status], [On], [HealthCheckExecutionId])
    VALUES
    (
        new.Name,
        new.Status,
        new.OnStateFrom,
        new.id
    );
END

[saddamhossain]

The UI is not working with .NET 7.0

[JonasJes]

@jsanjuan2016 I converted it to SQL and tried it out. While I got entries in the HealthCheckExecutionHistories, nothing appeared in the UI. I think you are triggering and logging from the wrong table. Instead of Executions you should use HealthCheckExecutionEntries as the name in HealthCheckExecutionHistories should match the health check entries

This worked for me as a temporary workaround:

CREATE TRIGGER after_update_executionentries ON HealthCheckExecutionEntries
   AFTER UPDATE
AS DECLARE @Name NVARCHAR(500), @Status INT, @On DATETIME2(7), @HealthCheckExecutionId INT, @Description NVARCHAR(MAX)
SELECT @Name = ins.Name FROM INSERTED ins;
SELECT @Status = ins.Status FROM INSERTED ins;
SELECT @On = GETDATE();
SELECT @HealthCheckExecutionId = ins.HealthCheckExecutionId FROM INSERTED ins;
SELECT @Description = ins.[Description] FROM INSERTED ins;
BEGIN
    SET NOCOUNT ON;
    IF UPDATE([Status])
    BEGIN
        INSERT INTO HealthCheckExecutionHistories
            ([Name], [Status], [On], [HealthCheckExecutionId], [Description])
        VALUES
            (
                @Name,
                @Status,
                @On,
                @HealthCheckExecutionId,
                @Description
        );
        PRINT 'HealthCheckExecutionEntries was updated and an event trigger was updated into HealthCheckExecutionHistories'
    END
END

[DeepWorksStudios]

i can comfirm this is still an issue today with the lastest versions