DEFRA / water-abstraction-team

Guides, info and issue management for the Water Abstraction Team
Other
1 stars 0 forks source link

Only log errors! #79

Closed Cruikshanks closed 1 year ago

Cruikshanks commented 1 year ago

We have an ongoing problem where the log files for our apps very quickly become massive. In the space of a week, we can see a repo like water-abstraction-returns generate a log file almost 10Gb in size!

Fortunately, we have a solution in place for production that truncates them on a regular basis. But because we switch off our non-prod environments that process is often blocked there, meaning the logs get out of hand and use up all available disk space. It also means our local environments can be extremely noisy and if we don't switch them off, they too can run into the same issue.

So, for our background web services, we are changing the hapi-pino config to be based on the existing WRLS_LOG_LEVEL env var.

We're also defaulting it to 'warn' in our non-prod and local environment so we'll only log request errors. This will also have an effect on the existing logging in the app. Now, only when it's logging an error will we see something.

Cruikshanks commented 1 year ago

All done! Not a permanent fix, but it will do until we have properly sorted our logging!