Modernize logger and implement logging standards

[timolegros]

Description

Our logging library (typescript-logging v0.6.4) is now 4 years old and inadequate for our needs. Specifically, issue https://github.com/hicommonwealth/commonwealth/issues/5230 describes current problems/bugs we face with our existing logger. In addition to these issues, we need to update our logging strategy so that we adhere to best practices such as properly using log levels (trace vs info vs warn vs error vs critical), logging only in certain environments (don't need to log in CI tests), and abstracting away from error reporting and aggregation in Rollbar (e.g. combine log.error and rollbar.error).

Required logger features/standards:

• Performant (should be lightweight and fast)
• Structured logs
  • This means logs are logged in JSON in production (can be pretty-printed for development) to aid in external parsing of logs on Datadog for increased visibility and metric tracking without expensive string parsing/searching
  • We should ensure that structured logs are cleanly represented in Rollbar and Datadog.
• Supports context fields
  • This allows us to add context via JSON properties to a log outside of the actual message (e.g. user ID or request URL)
• Log levels
• Log timestamps
• Log factory or inheritance
  • The logging package should allow us to create and organize loggers with various settings (log-level, attached context, etc).
• Log stack traces
• Supports log redacting to remove sensitive data from context (IP addresses, passwords, etc).
• Compatible with Typescript (i.e. is fully typed either by default or via an external types package).
• Browser-compatible to enable standard logging strategies across the entire stack
• Should be able to easily integrate with Rollbar

Project Owner

@timolegros

Engineering Requirements

• Update documentation on logging to ensure the logging standards we set as part of this issue are available to all (@gdjohnson for visibility)
• Migrate to a modern logging package such as Typescript-logging v2, Winston, Pino, or any package from the following list that matches our logging requirements/needs: https://betterstack.com/community/guides/logging/best-nodejs-logging-libraries/
• Configure logging levels on all of our environments (reduce unnecessary log volume on Datadog and in CI)
  • This can include both configuring the environment logging levels and the logging level of specific logs (for example logging every HTTP request as DEBUG instead of INFO)

Specifically, complete these tickets in order:

• [x] https://github.com/hicommonwealth/commonwealth/issues/6157
• [x] https://github.com/hicommonwealth/commonwealth/issues/5230
• [x] https://github.com/hicommonwealth/commonwealth/issues/6158
• [x] https://github.com/hicommonwealth/commonwealth/issues/6159
• [ ] https://github.com/hicommonwealth/commonwealth/issues/6160

Additional context

[timolegros]

Blocked by #6150

[gdjohnson]

@timolegros We don't have any logger documentation AFAICT, but now's a great time to start.

How are you thinking to handle? I'm always happy to hop on a call as we've done with Heroku & DataDog. (I shouldn't be as slow on the turnaround this time as I've been with DataDog—was very sick the second half of December, had to take some time.)

[timolegros]

Unblocked since #6150 won't be merged directly.

@gdjohnson Let's hop on a call once the 3rd ticket (#6158) is in progress since I will know more details about logging by then.