Creating an easy way of storing logs to different folders/files based on function location in Node.js

[TheTechChild]

Hello all, I am a noob to pino so forgive me for probably appearing like a dork. But I have a problem that I am hoping that pino can help me solve with my application. and the problem is this:

I have an application that runs an express app that operates at the root level of the application folder structure. The server is in charge of running background processes, responding to requests, kicking off cron-jobs, etc. I would like to log errors, info, and debugging information about those processes. which I know how to accomplish using the express-pino-logger library and the redirect operator for the command line. Accomplished by something akin to the following:

const express = require('express');
const pino = require('pino');
const expressPino = require('express-pino-logger');

const logger = pino({ level: process.env.LOG_LEVEL || 'info' });
const expressLogger = expressPino({ logger });

const PORT = process.env.PORT || 3000;
const app = express();

app.use(expressLogger);

app.get('/', (req, res) => {
  logger.debug('Calling res.send');
  res.send('Hello World');
});

app.listen(PORT, () => {
  logger.info('Server running on port %d', PORT);
});

and running the command:

 LOG_LEVEL=debug node index.js > ./Logs/info.log 2> ./Errors/errors.log

However, the complication comes in with the following desired behavior of the application. One of the jobs that this application is in charge of is getting data from many data sources on a daily basis (mostly getting data from apis, scraping web portals, websites, etc.). And some of these jobs are even on an hourly basis. These jobs and data acquisitions are very lengthy processes, so unfortunately I need to run multiple of these jobs at the same time, which you can imagine that the debugging logs if one of those jobs goes awry would be an insane jumble of logs from multiple sources. So piping the debugging msgs and errors into single files is obviously out of the question.

The ideal solution to this problem would be to pipe the msgs that are only on the server level into a single log file, and have a logs folder structure that can be separated into the different data acquisition types. Such as something like the following:

--Logs Parent Directory--
   --> Server
              errors.log, info.log, debug.log
   --> Web Portal 1 Scrape - Logs
              errors.log, info.log, debug.log
   --> Website 1 API - Logs
              errors.log, info.lgo, debug.log
......etc......

and then in the Web Portal 1 Scrape javascript file build a way to log directly into the Web portal 1 Scrape Folder in the Logs Parent Directory. As well as maybe noting to the server a single line if one of the jobs fails, so that I have a record that something happened for that job and I can dig into the logs for that specific job to see what occurred.

I hope that the problem is clear as I have described it, but I know that it encompasses a lot. If you need more information please let me know and I will provide more example code/whatever you need to be able to correctly answer the problem.

[TheTechChild]

UPDATE: I thought I would supplement this post with some more code and include some files as well as a picture of the folder directory so that more context can be provided.

the npm packages required for this code to work: npm install express body-parser nightmare fs-extra request-promise-native pino express-pino-logger pino-pretty

index.js:

const express = require('express');
const bodyParser = require('body-parser');
const pino = require('pino');
const expressPino = require('express-pino-logger');

const logger = pino({ level: process.env.LOG_LEVEL || 'info' });
const expressLogger = expressPino({ logger });

const PORT = process.env.PORT || 3000;
const app = express();

const swAPIPull = require('./modules/apiIntegrations/pullSWApi');
const googleScrape = require('./modules/scrapers/scrapeGoogle');

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.use(expressLogger);

app.get('/', (req, res) => {
  logger.debug('Calling res.send');
  res.send('Hello World');
});

app.post('/swAPI', (req, res) => {
  let startNum = req.body.startNum;
  let endNum = req.body.endNum;

  swAPIPull({startNum, endNum});
  /** I AM PURPOSEFULLY IGNORING THE PROMISE FROM swApiPull because the response is not necessary for the response to the user **/

  res.send('swAPI pull begun');
});

app.post('/google', (req, res) => {
  googleScrape();
  res.send('google scrape begun');
});

app.listen(PORT, () => {
  logger.info('Server running on port %d', PORT);
});

pullSwAPI.js:

const
path = require('path'),
rp = require('request-promise-native');
const writeToFile = require('../helpers/helpers').writeToFile;
const dataDirectory = path.join(__dirname, '..', '..', 'data', 'swapi', 'people.json');

module.exports = async ({startNum, endNum}) => {
  let baseURL = `https://swapi.dev/api/people`;
  for (let i = startNum; i < endNum; i++) {
    let response = await rp.get(`${baseURL}/${i}`)
      .then(response => {
        /** THIS IS WHERE I WOULD WANT TO INCLUDE LOGGING TO THE STAR WARS API LOGS/INFO or DEBUG**/
        return response;
      }).catch(e => {
        /** THIS IS WHERE I WOULD WANT TO INCLUDE LOGGING TO THE STAR WARS API LOGS/ERROR **/
      });

    let filePath = path.join(dataDirectory, 'people.json');

    await writeToFile({filePath, data: response});
  }
  return { successful: true };
};

scrapeGoogle.js:

const path = require('path');
const Nightmare = require('nightmare');
const dataDirectory = path.join(__dirname, '..', '..', 'data', 'google', 'data.json');
const delay = require('../helpers/helpers').delay;

module.exports = async () => {
  let nightmare = new Nightmare({ show: true });
  let url = 'https://www.google.com';

  let data = await nightmare.goto(url)
    .evaluate(() => {
      /** DO STUFF TO SCRAPE THE WEBSITE **/
    })
    .then(async response => {
      /** INCLUDE LOGGING TO THE GOOGLE SCRAPE / debug **/
    })
    .catch(e => {
      /** SEND ERROR TO THE GOOGLE SCRAPE LOGS / errors **/
    });
  await delay(10000);

  await nightmare.end();
};

helpers.js:

const fs = require('fs-extra');

module.exports = {
  writeToFile: async ({ filePath, data, minify = true }) => {
    let formattedData;
    if (minify) formattedData = JSON.stringify(data);
    else formattedData = JSON.stringify(data, null, 4);
    fs.outputFile(filePath, formattedData);
  },
  delay: (time) => {
    return new Promise(resolve => {
      setTimeout(() => {
        resolve();
      }, time);
    });
  }
};

[TheTechChild]

@jsumners Thank you for editing the original post to show the code clearly. I always forget how to do that on github.

Note to all following this post. I forgot to include bodyParser in the list of packages that you need to install from npm in order to get the app.posts to work as written. I have updated the code to include body-parser, and I have also updated the npm install command to include body-parser

[mcollina]

I would recommend you to send all the logs to the same file (or better sdout) and then post-process them in the structure you need.

In order to do so you can add as many properties to the newline delimited json:

log.info({ source: 'google' }, 'my message')
const child = log.child({ source: 'google' })
child.info('my message') // same as before

In this way you can write another program that would post-process your log files.

[TheTechChild]

That's an interesting idea @mcollina. I need to look at it. I am not yet completely familiar with the log.info and log.child syntax. I need to go read the documentation further to see how to properly use that.

I don't know if this is even possible or best practice to do this, but is it possible to use the command line when I start up the application to send the output into a program as opposed to a file? like for instance doing the following:node index.js > ./logs/my_log_processor_program.js

as opposed to:node index.js > ./logs/server/info.log which just outputs to the file info.log

I know that that is probably incorrect syntax, but I hope that the idea is clear of what I would like to accomplish.

Again, I am a complete noob at logging and I apologize for my naiveté. Like I said I was just promoted to this position, before this my main responsibilities were maintaining a front-end application and writing scrapers and api integrations on the side. I haven't had to create an application with good logging before. So I apologize again. Thank you all for your patience.

[mcollina]

you can do

node index.js | node another.js

[TheTechChild]

Sweet, I am glad I can redirect to another node program. That will make handling all these logs a lot easier. Thank you so much!

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.