search

aws / amazon-cloudwatch-agent

CloudWatch Agent enables you to collect and export host-level metrics and logs on instances running Linux or Windows server.
MIT License
445 stars 204 forks source link

Please make it possible to read from the Systemd journal. #382

Open rnhurt opened 2 years ago

rnhurt commented 2 years ago

Currently, the only way to get logging information out of Systemd services is to port the information to (r)syslog or use something like Datadog's Vector product. However, it would be great if we could reference Systemd journal just like we do with log files.

I propose that you add a new "logs" section named, systemd, as an addition to the existing files and windows_events sections.

lwoggardner commented 1 year ago

possibly related question. Has anyone tried reading from journald using a named pipe to a journalctl process?

Either multiple named pipes for different subsets of the journal, or just one and use filtering on the cloudwatch agent side?

MioG777829 commented 1 year ago

Important reference point - the Elastic Beanstalk AMIs which use AL2023 have rsyslog installed to create logs in the file system as it was before. The EB team is working around this by keeping legacy configuration in place. There is no firm AWS strategy (that is visible) to upgrade these approaches to extract information directly from systemd journal.

rdkls commented 8 months ago

I would've thought this would be very important feature for the agent; would think one of the most common use cases is someone spinning up an ec2 instance (now AL3 which uses only default journald) and wanting to easily get logs into CW. Is there a reason it's not?

andreaswittig commented 8 months ago

Would be so cool to be able to collect metrics from Amazon Linux 2023 with amazon-cloudwatch-agent as well.

ozbillwang commented 6 months ago

fixed with below commands,

To AWS Developers who maintain the CloudWatch Agent:

please make sure, when install CloudWatch Agent, it sets ACL on the related folder , subfolders and files under the target folders.

set ACL to existing files and sub-folders

setfacl -R -m u:cwagent:rx /var/log

and add ACL on folder /var/log, for any new files and subfolders created under it.

sudo setfacl -R -d -m u:cwagent:rx /var/log

Not sure, why they can't be run with together with -d and -m.

So now you can check the ACL if they are set properly

# getfacl /var/log
getfacl: Removing leading '/' from absolute path names
# file: var/log
# owner: root
# group: root
user::rwx
user:cwagent:r-x
user:splunk:r-x
group::r-x
mask::r-x
other::r-x
default:user::rwx
default:user:cwagent:r-x
default:user:splunk:r--
default:group::r-x
default:mask::r-x
default:other::r-x

# getfacl /var/log/messages
getfacl: Removing leading '/' from absolute path names
# file: var/log/messages
# owner: root
# group: root
user::rw-
user:cwagent:r-x
user:splunk:r--
group::---
mask::r-x
other::---

Notes

Splunk agent, for your reference, seems to know this issue, it automatically set ACL on /var/log properly after installation

grodriguezl commented 4 months ago

It's incredible that this isn't solved for a fresh AL2023 and I need to use syslog or change systemd files. What's the point of removing syslog if the new solution can't be integrated with cloudwatch agent?

cartwrightian commented 5 days ago

Can someone make clearer on the documentation that cloud watch agent is not compatible with amazon linux 2023 if you want to see system logs, save folks wasting time googling etc and eventually ending up here.