Using wildcard path for journal target

[dmitrievav]

The problem description

According to documentation https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#journal

journal settings block has a parameter path of type string which currently does not support wildcards.

level=error ts=2024-08-19T23:33:37.331758236Z caller=journaltarget.go:269 msg="received error reading saved journal position" err="failed to open journal in directory \"/var/log/journal/*\": no such file or directory"
level=error ts=2024-08-19T23:33:37.331837979Z caller=main.go:169 msg="error creating promtail" error="failed to make journal target manager: creating journal reader: failed to open journal in directory \"/var/log/journal/*\": no such file or directory"

On EKS node /var/log/journal directory contains subfolders to organize and store log data in a structured manner. Each subfolder typically corresponds to a unique identifier for a boot session or a specific machine ID, allowing for efficient log management and retrieval.

[root@ip-xx-xx-xx-xx /]# ls -alF /var/log/journal
total 4
drwxr-sr-x+ 4 root systemd-journal   86 Aug 18 16:18 ./
drwxr-xr-x  9 root root            4096 Aug 18 16:18 ../
drwxr-sr-x+ 2 root systemd-journal   28 Aug 18 16:18 ec2048eddce9d642c6a870d2ee447f4f/
drwxr-sr-x+ 2 root systemd-journal   28 Aug  7 20:54 ec22ede2efb295a808f4e640893255d9/
[root@ip-10-129-3-216 /]# ls -alF /var/log/journal/*/
/var/log/journal/ec2048eddce9d642c6a870d2ee447f4f/:
total 16388
drwxr-sr-x+ 2 root systemd-journal       28 Aug 18 16:18 ./
drwxr-sr-x+ 4 root systemd-journal       86 Aug 18 16:18 ../
-rw-r-----+ 1 root systemd-journal 16777216 Aug 20 09:30 system.journal

/var/log/journal/ec22ede2efb295a808f4e640893255d9/:
total 8196
drwxr-sr-x+ 2 root systemd-journal      28 Aug  7 20:54 ./
drwxr-sr-x+ 4 root systemd-journal      86 Aug 18 16:18 ../
-rw-r-----+ 1 root systemd-journal 8388608 Aug  7 20:58 system.journal

Name of subfolder cannot be known in advance and that is why configuring journal target without wildcard is impossible.

Possible solution

That would be necessary to allow parameter path to accept wildcards as in example below:

# helm_values_dev.yml for promtail

config:
  snippets:
    extraScrapeConfigs: |
      - job_name: "systemd-journal"
        journal:
          path: /var/log/journal/*

[Garbage4F]

Discovered this issue whilst looking for the same solution.

@dmitrievav for the current machine the boot ID is known at boot, so as a hacky work around you could use ExecStartPost= in the systemd unit for any nested machine's that get booted following that, this could populate your promtail .yaml with correct values

[JStickler]

Promtail is now considered “feature complete” and will be in a maintenance mode. Requests for new features should be made against Grafana Alloy, Grafana Labs’ distribution of the OpenTelemetry Collector.