Use a proper Logger instead of console.log

[johnagl]

Right now we're using console.log() to display relevant information in the terminal. Use something else such as https://www.npmjs.com/package/pino.

We should also look into making the linter complain if it sees console.log().

[resurl]

update for this: i have the pino logs set up for both docker and local machine (local is colored coded, its a little tricker to do this with docker but i assumed it wasn't a big deal)

but i'm reading more about logging and getting a bit confused lol. when it comes to development, it seems that logging in stdout is ok, but for production + docker, logging has to be handled a little differently.

for production, i see recommendations for outputting to json logs and for docker, i see that most people recommend staying away from logger libraries because of something called docker logging drivers / docker agents which do the same job for deploying with docker.

but since it works anyway, should i even worry about that right now?

[johnagl]

Oof... Okay I'm not very familiar with logging in general especially for production. @bobheadxi do you have any thoughts on this?

[bobheadxi]

@resurreccionl the advice you are finding might make more sense with some context:

1. Docker picks up stdout/stderr automatically, and keeps track of this output itself (which is how docker logs works on stopped containers). There is software out there that can manage this data for you asynchronously (known as log drivers in Docker, such as docker-gcp-logsdriver)
2. "Logging libraries" in other languages often refer to push-based logging - logs get recorded and pushed in-application to external services, in what is often called a "log sink". This is an absolutely terrible idea in Node - it is single-threaded, and spending valuable compute time on pushing logs is a no-go, which is why Pino works the way it does: it focuses on log formatting, leaving log delivery to what it calls transports such as pino-stackdriver (which I added some features to a while back hehe) which runs and delivers logs sidecar-style so as to not take up resources in your main applications.

How does this play into what you are trying to set up? tl;dr just use a log formatting library like Pino and don't worry about the rest yet - in dev, you can use the human-friendly output mode in your screenshots, and in prod use the JSON output mode. When it comes time to deliver your logs to an external service, look for something that can pick up JSON logs from Docker. You might not even have to do this, depending on your infrastructure - some VPS services treat all output on an instance as logs, and automatically exports that for you!

Now, why JSON logs? Logs play an important role in the debugging component in a topic called observability, which is how you triage problems once you know there is a problem (knowing if there is a problem is called monitoring) and the format of your logs should be designed help you out.

• Locally or in dev, log volume is low enough that you can grok your output at a glance and find useful messages quickly. The priority is human-readability: cute colors, nice spacing, all the nice stuff.
• In production, your traffic might generate huge volumes of logs, with multiple things happening at the same time, such that it might be impossible to find what you are looking for. This is where structured logs come in - in JSON format, you prioritize machine-readability, allowing you to query for specific types of logs that can save you a lot of time, such as:
  • get all logs associated with a request ID (id="abcde")
  • get all error-level logs (lvl="error")
  • get all messages generated by your database module (logger="db")
  • In more advanced usage, log fields can be used to generate metrics as well, which can give you the monitoring component of observability (such as alerts when too many errors happen)

How this all works out in practice: for example, in Pino you might create child loggers to encapsulate useful fields and context:

// per-module
const moduleLogger = logger.child({ logger: 'scraper' })

// per-request
const requestLogger = logger.child({ requestID: 'abcde' })

Then, you can have a "parent" logger from which all child loggers are derived that:

• enables pretty-printing in NODE_ENV=development
• sets log levels depending on your environment (for example, maybe you don't want all those debug logs in production)

You then pass these loggers around as parameters to maintain context in your log messages.

Happy to clarify more if needed!

---

ps

I love Pino, and my Go recommendation is [zap](https://github.com/uber-go/zap), alongside [shameless plug for my zap extensions library](https://github.com/bobheadxi/zapx) 🙃 I also worked on a lot of the [infrastructure and documentation behind Sourcegraph's observability stuff](https://about.sourcegraph.com/handbook/engineering/observability), and also did some [fun monitoring stuff at Riot Games](https://bobheadxi.dev/evaluable-expressions/)

[resurl]

@bobheadxi wow thanks so much, this helps a lot! i saw some of what you've mentioned in isolated bits and pieces while researching, but this really put it all together.

[johnagl]

That was a really good explanation of logging @bobheadxi and underscores the importance of logging in general. Thank you!