My goal is to create a pipeline that is built exclusively with tools I either already know, or am only learning because they provide added value to the project
Since one of my goals is to be able to scale wherever possible, I am trying to figure out how to do things in a more automated fashion. I am still manually parsing logs, and that is part of my problem, because a lot of my time spent looking at logs is just "looking for something to jump out at me."
If I implement some sort of "best practices" logging, then it would likely help highlight the things I am looking for.
Those are both seemingly related to node/node js. I know almost nothing about node at this point, though I know Justin and Bob both seem to use it a lot. I assume it is related to npm, but that may be erroneous. This article (https://blog.heroku.com/debug-node-applications) gave me some interesting ideas, so I will see if I can do a spike on seeing if I can use logging to understand what is going on by looking at what "best practices" with logging are for the specific aspects of my pipeline. I will likely also start by looking into k8s log parsing, which would help with my current task that is very overdue.
This task is done when I can define "logging at scale as it applies to my pipeline."
Since one of my goals is to be able to scale wherever possible, I am trying to figure out how to do things in a more automated fashion. I am still manually parsing logs, and that is part of my problem, because a lot of my time spent looking at logs is just "looking for something to jump out at me."
If I implement some sort of "best practices" logging, then it would likely help highlight the things I am looking for.
The two most likely contenders for this seem to be pino https://github.com/pinojs/pino or winston https://github.com/winstonjs/winston.
Those are both seemingly related to node/node js. I know almost nothing about node at this point, though I know Justin and Bob both seem to use it a lot. I assume it is related to npm, but that may be erroneous. This article (https://blog.heroku.com/debug-node-applications) gave me some interesting ideas, so I will see if I can do a spike on seeing if I can use logging to understand what is going on by looking at what "best practices" with logging are for the specific aspects of my pipeline. I will likely also start by looking into k8s log parsing, which would help with my current task that is very overdue.
This task is done when I can define "logging at scale as it applies to my pipeline."