Elastic Agent watch command always logs at the debug level in the elastic_agent data stream in Fleet

[amolnater-qasource]

Kibana Build details:

VERSION: 8.9.0 Snapshot
BUILD: 63944
COMMIT: 9d6846d9e259192dd968c503fac40c44d42c13b9

Host OS and Browser version: Windows & Linux, All

Preconditions:

1. 8.9.0 SNAPSHOT Kibana cloud environment should be available.
2. Few Agents should be installed.

Steps to reproduce:

1. Navigate to Fleet>Agents>Logs tab.
2. Set debug filter from the log level.
3. Observe debug level log line is generated: elastic_agent [elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'.

Expected: Debug log should not be generated on agent installation, when debug level is never set.

• Expected order: error>warning>info>debug

Screenshots:

What's working fine:

• This log line is also available for 8.8.1 agent.

Logs: elastic-agent-diagnostics-2023-06-09T09-15-49Z-00.zip

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[cmacknz]

This appears to be coming from the upgrade watcher logs that we now save properly.

~/Downloads/elastic-agent-diagnostics-2023-06-09T09-15-49Z-00 ···························· 03:55:50 PM
❯ cat logs/elastic-agent-7371b4/elastic-agent-watcher-20230609.ndjson
{"log.level":"debug","@timestamp":"2023-06-09T08:39:04.134Z","log.origin":{"file.name":"cmd/watch.go","file.line":68},"message":"update marker not present at 'C:\\Program Files\\Elastic\\Agent\\data'","ecs.version":"1.6.0"}

I believe this is because the watcher always logs at the debug level:

https://github.com/elastic/elastic-agent/blob/636bf0691844b177ba2ce55b35aef0498f161414/internal/pkg/agent/cmd/watch.go#L213-L214

I don't think users should have control of the watcher log level since it is critical to root causing upgrade failures, but this is also confusing. Probably we should be shipping these logs to their own data stream, where it would still ignore the log level but at least it would be obvious they aren't from the running agent process itself.

[michalpristas]

code changed, but issue is still present in managed scenario

[WiegerElastic]

code changed, but issue is still present in managed scenario

Would this be the reason I am seeing errors that aren't actually errors when upgrading Agents?

https://github.com/elastic/infosec/issues/16116#issuecomment-1903663422

[cmacknz]

Those are the output of the endpoint-security command used to upgrade the endpoint-security service, those are unconditionally logged at the error level. Probably we can reduce them to info when the command succeeds, I can't recall a situation where they were useful if the command didn't fail.