gordon-cs / gordon-360-api

The 360° Gordon Experience
11 stars 6 forks source link

Log every API Request/Response #968

Open amos-cha opened 1 year ago

amos-cha commented 1 year ago

Handles: https://github.com/gordon-cs/gordon-360-api/issues/940

For code explanation: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware/extensibility?view=aspnetcore-6.0 I can't explain as well as Microsoft's own documentation. The reason I've selected a factory based middleware activation is due to the requirement of dependency injection of our DbContext.

This is an incredibly hefty table (CCT.dbo.RequestResponseLog) and I've tried to minimize unnecessary data but the table is most definitely up for change and will probably need a trigger to clear the table on occasion. Merge to develop will be necessary for testing under load.

russtuck commented 1 year ago

I did a little searching for best practices in handling .NET logs. Here are some notes on what I found (with references at the bottom).

  1. They were all focused on using logs to debug errors in production. That's great, but it's not what we're doing here. Logging for debugging might include this, but would also include other things.

  2. Logging to a database is one of the top 3 options. Another is logging to a separate service that other things can listen to if they want the logs. And of course the other is log files.

We could adopt one of the log saving systems recommended in the references below, but then we'd also need to figure out how to get the data into a machine learning system. Or we can simply save it into the DB, where it's also easy to query it for ML.

My vote is to start logging to the database, and see how it goes. If the DB shows signs of stress as a result, we can disable it and then try another option.

References: https://michaelscodingspot.com/logging-in-dotnet/ https://stackify.com/csharp-logging-best-practices/ https://raygun.com/blog/c-sharp-logging-best-practices/

bennettforkner commented 1 year ago

All of our requests come through a proxy, so logging the source IP address is probably not going to be super helpful in our case. You would have to do something smarter to get the source IP from the client/UI side if you wanted it.

russtuck commented 2 months ago

Evan prefers a different approach (and has some good reasons), and started the work in #1036. But more work is needed to make that useful: