search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
19 stars 144 forks source link

Mixing json and non-json logs in elastic-agent-startup.log #4641

Open nick-benoit opened 6 months ago

nick-benoit commented 6 months ago

Currently cloud is expecting the non-json logs. As discussed with @cmacknz it seems like the best path forward is to convert the non-json logs to the standardized json format. It is unclear exactly when this started happening or if it has always been this way and just not noticed until now.

Example log entries from 8.12.2

Starting in elastic cloud mode
Preparing apm-server for legacy mode.
Starting legacy apm-server daemon as a subprocess./usr/share/elastic-agent/data/elastic-agent-de80b0/components/apm-server
Legacy apm-server daemon started.
Policy selected for enrollment:  policy-elastic-agent-on-cloud
{"log.level":"info","@timestamp":"2024-03-25T10:04:11.692Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":628},"message":"Spawning Elastic Agent daemon as a subprocess to complet
e bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-25T10:04:11.692Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":628},"message":"Spawning Elastic Agent daemon as a subprocess to complet
e bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-25T10:04:11.738Z","log.origin":{"file.name":"cmd/run.go","file.line":157},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"proces
s.pid":157,"agent.version":"8.12.2","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-25T10:04:11.738Z","log.origin":{"file.name":"cmd/run.go","file.line":157},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"proces
s.pid":157,"agent.version":"8.12.2","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-25T10:04:11.954Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":114},"message":"agent is not upgradable, not starting watcher","log":{
"source":"elastic-agent"},"ecs.version":"1.6.0"}
elasticmachine commented 6 months ago

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

cmacknz commented 6 months ago

It has always been this way as far as I can tell, we don't have an instance of the JSON logger in our container command implementation:

https://github.com/elastic/elastic-agent/blob/5fe8debfade598f0d3bd358bb35f7f69ba5b3664/internal/pkg/agent/cmd/container.go#L212-L253

None of that code has changed in years.

cmacknz commented 6 months ago

All logging is done with these two:

https://github.com/elastic/elastic-agent/blob/5fe8debfade598f0d3bd358bb35f7f69ba5b3664/internal/pkg/agent/cmd/container.go#L151-L157