sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint
https://sematext.com/logagent
Apache License 2.0
389 stars 79 forks source link

SC-16333 Extract fields from vercel syslog messages #325

Closed PabloB94 closed 1 year ago

PabloB94 commented 1 year ago

I added a regex to parse the kind of logs the user is asking about.

They are logs in the syslog format, vercel also provides an option to log in json format, which probably would be parsed correctly.

The regex assumes the same format the user reported, if the log doesn't match that format, it won't be parsed and it will default to the behavior we had until now

PabloB94 commented 1 year ago

Here's an example of the behavior of the parser:

For an input log:

{
  source: 'lambda',
  message: 'START RequestId: xxxxxxxx-da1f-48df-xxxx-b432102dd448 Version: $LATEST\n' +
    '2023-05-22T02:00:31.493Z\txxxxxxxx-da1f-48df-xxxx-b432102dd448\tINFO\t{\n' +
    "  level: 'info',\n" +
    "  message: 'Healthcheck',\n" +
    "  env: 'preview',\n" +
    '  healthy: true,\n' +
    '  prisma: { healthy: true, time: 0.011350886999629438, error: undefined },\n' +
    '  kinde: { healthy: true, time: 0.05071398300025612, error: undefined },\n' +
    '  hubspot: { healthy: true, time: 0.26839886800013485, error: undefined }\n' +
    '}\n' +
    'END RequestId: xxxxxxxx-da1f-48df-xxxx-b432102dd448\n' +
    'REPORT RequestId: xxxxxxxx-da1f-48df-xxxx-b432102dd448\tDuration: 275.22 ms\tBilled Duration: 276 ms\tMemory Size: 1024 MB\tMax Memory Used: 198 MB\t\n'
}

The output of the parser will be:

{
  requestId: 'xxxxxxxx-da1f-48df-xxxx-b432102dd448',
  version: '$LATEST',
  timestamp: '2023-05-22T02:00:31.493Z',
  logLevel: 'INFO',
  message: '{\n' +
    "  level: 'info',\n" +
    "  message: 'Healthcheck',\n" +
    "  env: 'preview',\n" +
    '  healthy: true,\n' +
    '  prisma: { healthy: true, time: 0.011350886999629438, error: undefined },\n' +
    '  kinde: { healthy: true, time: 0.05071398300025612, error: undefined },\n' +
    '  hubspot: { healthy: true, time: 0.26839886800013485, error: undefined }\n' +
    '}',
  duration: 275.22,
  billedDuration: 276,
  memorySize: 1024,
  maxMemoryUsed: 198,
  source: 'lambda'
}