search

pm4ml / mojaloop-connector

DEPRECATED by https://github.com/mojaloop/sdk-scheme-adapter
Apache License 2.0
0 stars 3 forks source link

enhancement: enable better log filtering #25

Open bushjames opened 3 years ago

bushjames commented 3 years ago
  1. Log volumes can be high if k8s health checks are frequent.
  2. Logs may be pretty printed json which is not ideal for grepping or processing with JSON tools like jq

Find a way to reduce log volume (possibly not log health checks?) and dont pretty print JSON; e.g. use one line per JSON object.

partiallyordered commented 3 years ago

Setting the space parameter of the logger config to 0 causes the logger to print one line per log event.

https://github.com/pm4ml/mojaloop-connector/blob/8695adaa5754f09705f061a934c278d798602c8e/src/index.js#L310

Note that this is exposed to the service config; setting LOG_INDENT=0 in the process environment will print a single line per log event.

One problem that may be encountered using jq for log filtering is if not every line printed is a json object. This can be worked around when printing ndjson or jsonl with the following (copied verbatim from my notes on the subject):

Given a log that contains pretty-printed json with some errors, we can add some delimiters and use --seq to instruct jq to ignore failed parses:

kubectl log -f deploy/some-application | sed 's/^{/'`echo "\036"`'{/' | jq --seq '.'

See --seq in the jq manual for more.

It would probably be wise to migrate mojaloop connector to use pino instead of the in-house logger. This reduces the maintenance burden. See the PR where this was done in finance portal backend: https://github.com/mojaloop/finance-portal-backend-service/pull/73 It might pay to check whether there are any additional changes to portal backend since then, for example to improve the formatting of stack traces or similar. This would also enable the use of pino-pretty, which can be used to pretty-print pino logs from a locally running service, or when run in k8s.