siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.89k stars 555 forks source link

Configurable extraTags when shipping kernel logs externally #9769

Open jtcressy opened 1 day ago

jtcressy commented 1 day ago

Feature Request

Addition of a new kernel arg talos.logging.kernel.extra-tag to apply extra tags to the logging destination configuration, similar to service logs.

We could format this new kernel arg similar to talos.environment so that multiple tags can be defined.

Also, if possible, include the support of templating into the value portion similar to talos.config with the options ${uuid} ${serial} ${mac} ${hostname} - this would enable us to include these identifiers as tags without hard-coding them.

Description

Metadata is distinctly lacking when shipping kernel logs to centralized logging systems. Ideally, I would want logs to include the source node's hostname at a minimum. As it stands, when one machine is having issues, it is almost impossible to identify it without opening the console of every machine and inspecting the logs manually.

If we had a way to include metadata via kernel arguments just like we can with service logs, this would solve the problem.

Example

Kernel cmdline for external logging with multiple templated tags talos.logging.kernel=udp://ingest.local:1234 talos.logging.extra-tag=hostname=${hostname} talos.logging.extra-tag=machine_serial=${serial} talos.logging.extra-tag=machine_uuid=${uuid} talos.logging.extra-tag=machine_mac_addr=${mac}