Logging

[GaProgMan]

Description

We should be logging all requests which come through to the API, tracing their journey through to when the relevant controller generates a response. This will be helpful in providing essential information when investigating bugs.

High-Level Proposed Solution

• Either use the built-in logging framework (from Microsoft) or SeriLog (or similar)
• Decide on which "sinks" to use for logging
  • File system?
  • Database?
  • Web service?
• Controllers, Services, and Datastores should all make logs when dealing with requests
• Each log line should contain:
  • DateTimeSpan with the server timezone and culture information
  • A request ID
  • a GUID generated at the controller's constructor perhaps?
  • or the HttpContext.TraceIdentifier
    • See: Logging HTTP context in ASP.NET Core for more information
    • As highlighted in the above blog post, request and response logging is likely best handled in middleware
  • Request details (only applicable at controller level)
  • IP address (only applicable at controller level)

Considerations

• We need to think of a solid way to test this in integration tests
  • Do we need to test this in integration tests? Or can we just test this in unit tests?
• We need to consider the security of the logs
  • Request data could (if we decide) contain IP addresses
  • We would need to decide whether we consider this PII
  • Similarly with User-Agent string
  • Perhaps, for now, we don't log this
• Where are we storing the logs?
  • Local development could be the local file system, but we would need to remember to clear them often
  • Perhaps a controller action which developers can call to clear the logs
  • Action must require a passphrase; stored in User Secrets
  • Action must respond only when in Development
  • Where should we store logs when in production?

Requirements

• Logging is provided throughout all parts of the API
• Logs should allow for a user request to be traced all the way to response generation

[GaProgMan]

Consider using a SIEM for logging, rather than logging out to a flat file or database.

Doing this will allow us to collect audit logs and, over time, increase the security of the system by setting up alerts for things like "too many requests", etc.

Consider investigating whether there are any SIEM offerings that can be containerised, and potentially use one for LOCAL DEV ONLY. That way contributors can run a single docker-compose (or similar) command to set everything up locally when wanting to work on or investigate something.