Added Serilog logging support to the project

[yogendrasgautam]

Implemented the logging for API and considered below points

• Used Serilog library
• Logs are being written to console, file and Serilog sink Seq server(http://localhost:5341/). This can be controled from appsettings.
• Log files are generated in Logs folder with rolling interval of day.
• Logs are ingested in serilog Seq sink for better visibility and dashboard support.
• To create Serilog sink Seq server, code to start the docker container (run --rm -e ACCEPT_EULA=y -p 5341:80 datalust/seq) is added in Program.cs. This code will execute only once while first time app deployment if there is no server available for serilog logs.
• Unit tests are not added for now, We can validate the approach for logging and if its cofirmed will proceed with adding test cases.

The changes can be deployed to staging and tested. Sanitization part is pending.

Points to discuss/consider

• To sanitize the response logging, Instead of logging inside middleware we would have to log the response in each controller action.
• We can sanitize the response log in middleware if we mainatain some property names that need to be masked or removed.

Please let me know your preference ?

Tasks:

1. Design:

   • [x] Define log formats and structures to ensure consistency and readability.
   • [x] Plan for log storage, retention, and access, considering scalability and security.

2. Backend Development:

   • [x] Integrate Microsoft.Extensions.Logging into the API codebase.
   • [x] Configure logging providers (e.g., console, file, Azure Application Insights) as needed.
   • [x] Implement logging for incoming API requests, capturing HTTP method, endpoint, headers, parameters, and request body.
   • [x] Implement logging for API responses, capturing status code and response body.
   • [x] Implement error logging to capture and log exceptions and stack traces.
   • [x] Add performance logging to measure and record request processing times.
   • [x] Ensure sensitive information is masked or omitted in the logs.
   • [x] Configure log levels and ensure the ability to adjust them as needed.

3. Testing:

   • [ ] Conduct tests to verify that all relevant information is being logged correctly.
   • [ ] Test the performance impact of logging to ensure it remains minimal.
   • [ ] Validate that sensitive information is properly masked or omitted from the logs.
   • [ ] Ensure that logs are stored securely and are accessible only to authorized personnel.
   • [ ] Test log retrieval and analysis to ensure logs are useful for monitoring and debugging.

4. Deployment and Monitoring:

   • [ ] Deploy the logging solution in a staging environment and monitor its performance and accuracy.
   • [ ] Make any necessary adjustments based on initial deployment feedback.
   • [ ] Deploy the logging solution to the production environment.
   • [ ] Set up alerts and monitoring based on log data to proactively identify and address issues.

[yogendrasgautam]

@NicolasBuscarini Do you have staging environment, where we can test logging solution? Docker service should be running in server.

[yogendrasgautam]

To prevent logging the sensitive response body used traceResponseBody boolean header with request.

By Default if header is not provided, we will trcae the request and response body.

[NicolasBuscarini]

@NicolasBuscarini Do you have staging environment, where we can test logging solution? Docker service should be running in server.

At the moment, no. I had issues with billing on Azure. I was supposed to try creating the services for free, but I haven't managed to do it yet.

Regarding the Deployment and Monitoring tasks and some of the Testing tasks, you might not be able to complete all of them. However, it's not a problem as we can handle them once I resolve the Azure issue.