search

amazonlinux / amazon-linux-2023

Amazon Linux 2023
https://aws.amazon.com/linux/amazon-linux-2023/
Other
500 stars 37 forks source link

[Bug] - rsyslog.service: Failed with result 'core-dump'. #736

Open crickc opened 5 days ago

crickc commented 5 days ago

Describe the bug Rsyslog sometime fail with status=7/BUS. Seems to happen randomly on almost all my EC2 instances and to relate to the systemd journal file rotation.

Expected behavior Should not crash.

Additional context 2024-06-17T07:38:56.454419+00:00 an-00030 systemd-journald[813]: Data hash table of /var/log/journal/ec2d15852ecc75d948b2139b7872ba60/system.journal has a fill level at 75.0 (173467 of 231288 items, 58720256 file size, 338 bytes per hash table item), suggesting rotation. 2024-06-17T07:38:56.454797+00:00 an-00030 systemd-journald[813]: /var/log/journal/ec2d15852ecc75d948b2139b7872ba60/system.journal: Journal header limits reached or header out-of-date, rotating. 2024-06-17T07:38:56.454985+00:00 an-00030 rsyslogd[1481]: imjournal: journal files changed, reloading... [v8.2204.0-3.amzn2023.0.4 try https://www.rsyslog.com/e/0 ] 2024-06-17T07:38:56.456327+00:00 an-00030 rsyslogd[1481]: imjournal: journal files changed, reloading... [v8.2204.0-3.amzn2023.0.4 try https://www.rsyslog.com/e/0 ] 2024-06-17T07:38:56.456385+00:00 an-00030 rsyslogd[1481]: imjournal: journal files changed, reloading... [v8.2204.0-3.amzn2023.0.4 try https://www.rsyslog.com/e/0 ] 2024-06-17T07:38:56.456429+00:00 an-00030 rsyslogd[1481]: imjournal: journal files changed, reloading... [v8.2204.0-3.amzn2023.0.4 try https://www.rsyslog.com/e/0 ] 2024-06-17T07:38:56.477531+00:00 an-00030 systemd[1]: Created slice system-systemd\x2dcoredump.slice - Slice /system/systemd-coredump. 2024-06-17T07:38:56.519523+00:00 an-00030 systemd[1]: Started systemd-coredump@0-1492626-0.service - Process Core Dump (PID 1492626/UID 0). 2024-06-17T07:38:56.600510+00:00 an-00030 systemd-coredump[1492628]: Process 1481 (rsyslogd) of user 0 dumped core. 2024-06-17T07:38:56.609424+00:00 an-00030 systemd[1]: rsyslog.service: Main process exited, code=dumped, status=7/BUS 2024-06-17T07:38:56.625570+00:00 an-00030 systemd[1]: rsyslog.service: Failed with result 'core-dump'. 2024-06-17T07:38:56.627888+00:00 an-00030 systemd[1]: rsyslog.service: Consumed 15min 52.377s CPU time. 2024-06-17T07:38:56.629027+00:00 an-00030 systemd[1]: systemd-coredump@0-1492626-0.service: Deactivated successfully. 2024-06-17T07:38:56.860793+00:00 an-00030 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 1. 2024-06-17T07:38:56.861950+00:00 an-00030 systemd[1]: Stopped rsyslog.service - System Logging Service. 2024-06-17T07:38:56.862039+00:00 an-00030 systemd[1]: rsyslog.service: Consumed 15min 52.377s CPU time. 2024-06-17T07:38:56.918826+00:00 an-00030 systemd[1]: Starting rsyslog.service - System Logging Service... 2024-06-17T07:38:56.937142+00:00 an-00030 systemd[1]: Started rsyslog.service - System Logging Service. 2024-06-17T07:38:56.937228+00:00 an-00030 rsyslogd[1492640]: [origin software="rsyslogd" swVersion="8.2204.0-3.amzn2023.0.4" x-pid="1492640" x-info="https://www.rsyslog.com"] start

Probably related to https://bugzilla.redhat.com/show_bug.cgi?id=2151092

zcobol commented 5 days ago

The imjournal module (what provides access to the systemd journal) is crashing rsyslogd after systemd-journald is rotating some journals. See /etc/rsyslog.conf. You might benefit from using systemd-journal-remote insteaad to forward journal entries over the network, or disable the imjournal module.