search

NixOS / infra

NixOS configurations for nixos.org and its servers
MIT License
230 stars 95 forks source link

Log aggregation #344

Open mweinelt opened 8 months ago

mweinelt commented 8 months ago

Is your feature request related to a problem? Please describe.

Querying the journal on rhea.nixos.org is annoying when it is under load. Just grepping through the last 24 hours takes a long time, and journalctl blocks a whole core, likely due to iowait.

Describe the solution you'd like

Stream the logs elsewhere, where we can store and query them.

Could be a simple as remote syslog, but could also be loki/promtail and allow integration and exposure via Grafana.

Describe alternatives you've considered

Additional context

n/a

RaitoBezarius commented 7 months ago

Relevant to the context: netdata is working on a bunch of systemd-journald optimizations. (I unfortunately lack of time to detail but I will comment when I have more time with precise details.)