Customize the jsonPayload object sent to GCP

[ehenon]

Hi everyone,

import winston from 'winston'
const { LoggingWinston } = require('@google-cloud/logging-winston')

const logger = winston.createLogger({
  transports: [
    new LoggingWinston({
      level: process.env.LOG_LEVEL,
      logName: process.env.GCP_LOG_NAME
    })
  ]
})

logger.info('Test log message', { stream: 'Stream name', feature: 'Feature name', labels: { plugin: 'Plugin name', key: '1#1#1' } })

Using this code snippet, I'm able to get this kind of log on GCP side:

{
  insertId: ".............BHAcE_Bz2t_Gx"
  jsonPayload: {
    message: "Test log message"
    metadata: {
      stream: "Stream name"
      feature: "Feature name"
    }
  }
  resource: { 2 }
  timestamp: "2020-11-06T12:45:12.197000026Z"
  severity: "INFO"
  labels: {
    plugin: "Plugin name"
    key: "1#1#1"
  }
  logName: "projects/...../logs/gcp_log_test"
  receiveTimestamp: "2020-11-06T12:51:10.945970109Z"
}

What I'm trying to do is to have control on the jsonPayload object in order to be able to put the different fields I want without having the extra metadata object, like this way:

{
  insertId: ".............BHAcE_Bz2t_Gx"
  jsonPayload: {
    message: "Test log message"
    stream: "Stream name"
    feature: "Feature name"
  }
  ...
}

I have tested different ways of logging, by entering myself the jsonPayload object explicitly (as we can do for labels object for example), or by trying to pass the original message as an object but I can't find a way to achieve my goal. Is this possible?

[0xSage]

You can use other nodejs logging libraries to achieve this effect. See https://github.com/googleapis/nodejs-logging

This library is more opinionated about how jsonPayloads are structured to conform with Winston.

Can you explain more about your use case? Why do you need all the values inside of jsonPayload instead of in say label or in metadata?

And would you still find the following helpful? If so, I can look into extending this library such that you can log JSON objects in jsonPayload.message.

jsonPayload: {
  metadata: {0}
  message: "{foo: "foo", bar: "bar"}"
}

[ehenon]

Hello and thank you for your quick answer. Regarding my use case, in fact the Common Digital Platform team of the company I work for has defined standards for the logs of all the company's apps. In these conventions, it is specified that on the GCP side, all JSON fields must be in the jsonPayload object, exactly at the same level as the message.

No need to look to expand the library so that we can log JSON objects directly in the jsonPayload.message as this would not be suitable for our conventions either. I'll see what I can do with googleapis/nodejs-logging! 😉

[0xSage]

Great. Do keep us posted on any issues. Happy to troubleshoot them with you

[zkreutzjanz]

I believe this actually does have an important use case. To automatically catch errors from logging for error reporting a type can be set under jsonPayload. You cannot currently do this with because of this implementation. See: https://cloud.google.com/error-reporting/docs/formatting-error-messages#log-text