Update approach to logging in connector component

artem-shelkovnikov commented 3 weeks ago

Closes https://github.com/elastic/connectors/issues/2792

Multiple logging-related changes and a small bugfix bundled together:

Now Agent Python component and Connectors hosted by it will log in ECS format. This will make agent/fleet parse logs better and provide better UX (see screenshots later)
Logging is actually consistently implemented in the agent component. There's a root logger and each file creates a child logger. Changes for root logger are automatically bubbled to child loggers which is nice
Added logging into all agent python component files
Python Agent Component now reacts to changes in log_level and properly bubbles log level changes to connectors as well
Fixed a bug when you change only log_level for fleet policy for the connector and it corrupts the configuration

Screenshots of the change in the Fleet UI:

[x] this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
[x] this PR has a meaningful title
[x] this PR links to all relevant github issues that it fixes or partially addresses
[x] if there is no GH issue, please create it. Each PR should have a link to an issue
[x] this PR has a thorough description
[ ] Covered the changes with automated tests
[x] Tested the changes locally
[x] Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)

jedrazb commented 1 week ago

For some reason python elastic agent logs have error level

artem-shelkovnikov commented 1 week ago

You can update requirements.txt to point to my branch artem/update-component-logger for the python elastic agent and see it yourself :)

github-actions[bot] commented 4 days ago

Status	Branch	Result
✅	8.x	https://github.com/elastic/connectors/pull/2847

This backport PR will be merged automatically after passing CI.