Multi log-table support?

[sommmen]

Hiya,

I've got the following setup, where i have 2 webapis and 1 hangfire backouground service running. They all log to different tables with serilog's MSSQL sink:

I'd love if the ui would support multiple services in some way.

I imagine something like:

        services.AddSerilogUi(options => options.UseSqlServer(_defaultConnectionString, "WebApi", "AppLogs"));
        services.AddSerilogUi(options => options.UseSqlServer(_defaultConnectionString, "AuthApi", "AppLogs"));
        services.AddSerilogUi(options => options.UseSqlServer(_defaultConnectionString, "Hangfire", "AppLogs"));

      ...
     app.UseSerilogUi(x => x.Endpoint("/applogs/webapi"));
     app.UseSerilogUi(x => x.Endpoint("/applogs/authapi"));
     app.UseSerilogUi(x => x.Endpoint("/applogs/hangfire"));

But that doesn't work since UseSqlServer registers a singleton that is used for any SqlServerDataProviders:

((ISerilogUiOptionsBuilder) optionsBuilder).Services.AddSingleton<RelationalDbOptions>(implementationInstance);.

I could also use one single logging table for all services, but then i'd need to be able to filter per application (an enrichter and a way to filter a spec. log property?).

Any ideas?

[mo-esmp]

Hey @sommmen, About your suggestion, I don't know how to map app.UseSerilogUi(x => x.Endpoint("/applogs/webapi")) to options.UseSqlServer(_defaultConnectionString, "WebApi", "AppLogs"), we need to find a way.

However, multiple log tables can be supported by adding array of log tables:

services.AddSerilogUi(options => options.UseSqlServer(_defaultConnectionString, "WebApi", ["WebApi", "AppLogs", ...]));

And in UI by choosing a log table from a drop-down list, logs can be filtered.

[sommmen]

Well anyways being able to filter on specific log properties would be extremely useful and it should not be too hard to implement (MySql / Tsql provders would be a simple where statemetn, and im not familiar with either elastic search or mongodb but i assume the same can be done)

About your suggestion, I don't know how to map app.UseSerilogUi(x => x.Endpoint("/applogs/webapi")) to options.UseSqlServer(_defaultConnectionString, "WebApi", "AppLogs"), we need to find a way.

I'm also a bit clueless on that part. I think it will be easier to just always have a single endpoint for the logviewer, and then a dropdown to select any tables:

services.AddSerilogUi(options => options.UseSqlServer(_defaultConnectionString, "WebApi", ["WebApi", "AppLogs", ...]));
...
app.UseSerilogUi(x => x.Endpoint("/SomeCustomEndpoint"));

Maybe would be cool if you could register multiple providers. We could do this in the backend with a simple SeriLogUiOptionsFactory that would return options for the provider(s). Just a simple key/value dict, where the key would be the SchemaName+Table - the same key you'd see in a dropdown.

Let me know if you have time to work on something like this - i'll try to find some time and perhaps make a pr but i'm quite busy in the near future.

[mo-esmp]

Could you please provide some codes on how to setup Serilog to log data in multiple tables in an ASP.NET Core application?

[sommmen]

Could you please provide some codes on how to setup Serilog to log data in multiple tables in an ASP.NET Core application?

Here's a sample project to emulate the situation, one console app with 3 mssql sinks all pushing to 3 different tables.

SerilogUiMultiDbSample.zip

using System.Diagnostics;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Serilog;
using Serilog.Debugging;
using Serilog.Events;
using SerilogUiMultiDbSample;

SelfLog.Enable(msg =>
{
    Console.WriteLine(msg);
    Trace.WriteLine(msg);
    Debug.WriteLine(msg);

    if (Debugger.IsAttached)
        Debugger.Break();
});

Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Debug()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Information)
    .Enrich.FromLogContext()
    .WriteTo.Console()
    .CreateLogger();

try
{
    var host = Host
        .CreateDefaultBuilder()
        .ConfigureServices(services => { services.AddHostedService<TestBackgroundService>(); })
        .UseSerilog((context, loggerConfig) =>
        {
            loggerConfig
                .ReadFrom
                .Configuration(context.Configuration);
        })
        .Build();

    host.Run();
}
catch (Exception ex)
{
    Log.Error(ex, "Caught ex");
}
finally
{
    Log.CloseAndFlush();
}

using System.Numerics;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace SerilogUiMultiDbSample
{
    internal class TestBackgroundService : BackgroundService
    {
        private ILogger<TestBackgroundService> _logger;

        public TestBackgroundService(ILogger<TestBackgroundService> logger)
        {
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            _logger.LogInformation("Starting right now!");

            while (!stoppingToken.IsCancellationRequested)
            {
                await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken: stoppingToken);

                var rand = new Random();

                using (_logger.BeginScope(new Dictionary<string, object>()
                       {
                           {"ARandomPropertyForSerilog", rand.Next(0, 420)},
                           {"DateTimeNow", DateTime.Now}
                       }))
                {
                    _logger.LogInformation("{DateTimeString} Big ben will bong {times} times.", DateTime.Now.ToString("T"), DateTime.Now.Hour);
                }
            }
        }
    }
}

{
  "ConnectionStrings": {
    "DefaultConnection": "Server=localhost\\SQLEXPRESS;Database=Test;User Id=sa;Password=opg;Trusted_Connection=False;Encrypt=False;MultipleActiveResultSets=True"
  },
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.Hosting.Lifetime": "Information",
        "System": "Warning",
        "System.Net.Http.HttpClient": "Warning",
        "Hangfire": "Warning"
      }
    },
    "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.MSSqlServer" ],
    "Enrich": [ "FromLogContext" ],
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {SourceContext}{NewLine}{Exception}"
        }
      },
      {
        "Name": "MSSqlServer",
        "Args": {
          "connectionString": "DefaultConnection",
          "sinkOptionsSection": {
            "tableName": "WebApi",
            "schemaName": "AppLog",
            "autoCreateSqlTable": true
          }
        }
      },
      {
        "Name": "MSSqlServer",
        "Args": {
          "connectionString": "DefaultConnection",
          "sinkOptionsSection": {
            "tableName": "AuthApi",
            "schemaName": "AppLog",
            "autoCreateSqlTable": true
          }
        }
      },
      {
        "Name": "MSSqlServer",
        "Args": {
          "connectionString": "DefaultConnection",
          "sinkOptionsSection": {
            "tableName": "Hangfire",
            "schemaName": "AppLog",
            "autoCreateSqlTable": true
          }
        }
      }
    ]
  }
}

EDIT: Oops - thats a normal net core project - not ASP. I read over that. The exact same code can be applied to an asp net core project though - is this ok for you?

[sommmen]

SidenoteI: there is the named options configurator which we could possibly use.

https://andrewlock.net/configuring-named-options-using-iconfigurenamedoptions-and-configureall/

However:

Note that the GetPublicWebhookUrl() method is synchronous, not async. Options configuration occurs inside a DI container when constructing an object, so it's not a good place to be doing asynchronous things like calling remote end points. If you find you need this capability, consider using other patterns such as a factory object instead of Options.

[sommmen]

Small bump / reminder to look at this (for both myself and others ^^).

At the moment i'm still having to query the db manually, using this would be superb.

[mo-esmp]

Hey @sommmen Thanks for reminding this and sorry about the delay. Past couple of month I was really busy and also @followynne refactored the UI. I think now it's the proper time to implement this feature, however, first I want to ask you about contributing on the backend side if you mind and have time. You can start by creating a feature from dev branch and together implement that.

[sommmen]

Hey @sommmen Thanks for reminding this and sorry about the delay. Past couple of month I was really busy and also @followynne refactored the UI. I think now it's the proper time to implement this feature, however, first I want to ask you about contributing on the backend side if you mind and have time. You can start by creating a feature from dev branch and together implement that.

No worries mate. I'm busy myself but i got about 4 hours of company R&D on fridays so if there are no accidents i can fill in some time there.

Will have to think of a nice way to integrate multible configs with the DI however, this week i tried to do something similar where i configured a bunch of provider services with the same interface, but then it really shows that MS DI is a very basic DI implementation (as designed).

[followynne]

Hi @sommmen cc @mo-esmp

Do you think something similar to https://github.com/followynne/serilog-ui/tree/feat/multi-table-reads could work? I tried this simple implementation that would work with the UI dropdown you mentioned in an earlier comment. I kept the current configuration as easy as possible, as it's just a test, but I think the string[] could be expanded with overrides to match other use cases:

• it could become an IEnumerable, to bind different configs objects in depth
• it could become a separate interface, that the user implements to create dynamic configs (ex: reading a dynamic source to retrieve connection strings, and table names... even after the service registration)

I gave a look at the IOptions registration, which is interesting but it would require creating different serilog-ui pages. If you think it would be a better option to create 1 separate page than choosing the queried table from a dropdown, we could explore it...

Let me know!

[sommmen]

Hi @sommmen cc @mo-esmp

Do you think something similar to https://github.com/followynne/serilog-ui/tree/feat/multi-table-reads could work? I tried this simple implementation that would work with the UI dropdown you mentioned in an earlier comment. I kept the current configuration as easy as possible, as it's just a test, but I think the string[] could be expanded with overrides to match other use cases:

• it could become an IEnumerable, to bind different configs objects in depth
• it could become a separate interface, that the user implements to create dynamic configs (ex: reading a dynamic source to retrieve connection strings, and table names... even after the service registration)

I gave a look at the IOptions registration, which is interesting but it would require creating different serilog-ui pages. If you think it would be a better option to create 1 separate page than choosing the queried table from a dropdown, we could explore it...

Let me know!

Hiya,

I myself was thinking more in the line of registering multiple providers or services, i don't feel like a provider should be able to handle multiple tables, its only there to provide data.

Just cloned the repo so ill take a stroll through the code, your solution seems very easy to implement so perhaps thats a good way to move forward.

2 unrelated remarks:

• First time building the app failed, i had to run npm install outside vs and then it seemed to work. unfortunately npm kept some files hostage so the build did not complete untill i closed the console host but hey thats a common issue so i recognized it quickly.
• You refactored the method UseMySqlServer its a public method and part of the lib interface so i'd suggest creating an override or adding optional properties so you don't break the interface and library consumers dont have to update code unnecesarily - i understand this is just a proto ofc.

[followynne]

Your point is good, I was thinking of the Provider as a "parameters receiver" with the selected table as a new parameter but table name is more of a configuration info.

Eventually, if we want something quick, we could develop both ideas - the simplest one will be available soon and the other can take its time to be ready. At the end it will be up to the consumer to choose one 😁

(ps I agree with you on the 2nd remark)

[sommmen]

Your point is good, I was thinking of the Provider as a "parameters receiver" with the selected table as a new parameter but table name is more of a configuration info.

Eventually, if we want something quick, we could develop both ideas - the simplest one will be available soon and the other can take its time to be ready. At the end it will be up to the consumer to choose one 😁

(ps I agree with you on the 2nd remark)

I'm running out of time unfortunately. Getting the IDataProvider into SeriLogUiMiddleWare is easy, but being able to get the right IDataProvider based on an Options class is hard. I've done this in my own project with generics, ill try and wire that up in here and see if it is a good fit.

That would look something like this for consumers:

.UseSqlServer(Configuration.GetConnectionString("DefaultMsSqlConnection"), "Logs"))
.UseMySqlServer(Configuration.GetConnectionString("DefaultMySqlConnection"), "Logs"))

Im now running into an issue where using that i'd have to inspect all options in a factory - and i'd rather only touch the one i need. Thinking i'd need to keep a dict of options. Ill continue this when i have more time unless you'd wanna go ahead and move on with the other solution but for me there is no hurry.

[followynne]

[ ... ] Im now running into an issue where using that i'd have to inspect all options in a factory - and i'd rather only touch the one i need. Thinking i'd need to keep a dict of options. Ill continue this when i have more time unless you'd wanna go ahead and move on with the other solution but for me there is no hurry.

What do you think of replacing the internal IDataProvider registration with something similar to:

// src\Serilog.Ui.MsSqlServerProvider\Extensions\SerilogUiOptionBuilderExtensions.cs, ln 43-45
((ISerilogUiOptionsBuilder)optionsBuilder).Services.AddScoped<IDataProvider, SqlServerDataProvider>(factory =>
{
    return new SqlServerDataProvider(relationProvider);
});

I was able to implement 2 versions of the same provider, that are reachable on different endpoints:

// src\Serilog.Ui.Web\Extensions\ApplicationBuilderExtensions.cs, ln 34 - 36
var providerServices = scope.ServiceProvider.GetServices<IDataProvider>();
[ ... ]
return applicationBuilder.UseMiddleware<SerilogUiMiddleware>(uiOptions, providerServices.FirstOrDefault(p => p.Name == uiOptions.SerilogName));

// in SampleWebApp.Startup
services.AddSerilogUi(options => options
  .UseSqlServer(Configuration.GetConnectionString("DefaultConnection"), "Logs")
  .UseSqlServer(Configuration.GetConnectionString("DefaultConnection"), "Logs2"));

[ ... ]
app.UseSerilogUi(p => { p.RoutePrefix = "logs2"; p.SerilogName = "Logs2"; });
app.UseSerilogUi(p => { p.SerilogName = "Logs"; });

The downside will be that the IDataProvider needs to implement a new Name field, to distinguish it when multi-registering them.
(if you want to take a look let me know, I'll cleanup the code and share the branch)