[Bug] - location of systemd journal changed

[daniejstriata]

Describe the bug I have an older installation of al2023 that I installed late last year and upgraded to the latest release. I can see a /var/log/journal directory and /run/log/journal but on an al2023 I installed yesterday (20240117) I don't have the /var/log/journal anymore. I find only /run/log/journal.

Is that change intended? I create a separate volume for logs and have not noticed the journal not being in /var/log anymore until now.

[nmeyerhans]

The /var/log/journal directory is created by the systemd package during execution of its %post scriptlet. That means that it is created during image creation, and exists in the base filesystem of the AMI.

The /run/log/journal directory is created at runtime and is used for ephemeral buffering before syncing to the permanent location in /var/log.

I've confirmed that /var/log/journal exists in the 20240117 AMIs. Can you be specific about where you're seeing this? AMI IDs will be helpful.

[daniejstriata]

The details of one of these are:

ami-089b0d542946462ed
al2023-ami-2023.3.20240108.0-kernel-6.1-x86_64

The Instances are hosted in Cape Town. [af-south-1] As you can see /var/log/journal is missing:

[nmeyerhans]

The /var/log/journal directory is present as expected on AMI ami-089b0d542946462ed.

[ec2-user@ip-172-31-5-213 ~]$ uptime
 14:14:06 up 0 min,  1 user,  load average: 1.00, 0.36, 0.13
[ec2-user@ip-172-31-5-213 ~]$ systemctl is-system-running 
running
[ec2-user@ip-172-31-5-213 ~]$ rpm -q system-release
system-release-2023.3.20240108-0.amzn2023.noarch
[ec2-user@ip-172-31-5-213 ~]$ ec2-metadata --ami-id
ami-id: ami-089b0d542946462ed
[ec2-user@ip-172-31-5-213 ~]$ stat /var/log/journal
  File: /var/log/journal
  Size: 46              Blocks: 0          IO Block: 4096   directory
Device: 10301h/66305d   Inode: 8566572     Links: 3
Access: (2755/drwxr-sr-x)  Uid: (    0/    root)   Gid: (  190/systemd-journal)
Context: system_u:object_r:var_log_t:s0
Access: 2024-01-05 21:59:45.240915486 +0000
Modify: 2024-01-23 14:13:11.829999978 +0000
Change: 2024-01-23 14:13:12.079999977 +0000
 Birth: 2024-01-05 21:58:05.004133679 +0000

You can see from the Birth field in the stat output that the directory was creating during the image generation process. So there would seem to be something else happening on your instances that's causing it to get lost. You mention that you set up a custom volume for your logging. The issue may be related to the logic behind that process.

[daniejstriata]

Thanks for having a look. I'll close this. I agree that /var/log/journal got lost when they moved in the new volume to mount /var/log on.

[daniejstriata]

I added the missing directory and rebooted the host. Journald is now writing there again.