systemd doesn't read logs from persistent storage by default

[orgads]

Bug Report

Describe the bug If journald logs are stored as volatile (/run/log/journal), systemd input plugin reads the logs correctly.

But if persistent storage is used (/var/log/journal), no logs are read.

To Reproduce

• Steps to reproduce the problem: Use the following configuration on a machine that has the logs stored in /var/log/journal.

  
  [INPUT]
  Name              systemd
  Tag               systemd
  Lowercase         On
  Strip_Underscores On
  DB                /fluent-bit/db/cursor.db

[OUTPUT] Name stdout Match *

**Expected behavior**
Logs should be read from `/var/log/journal` if it exists.

**Your Environment**
* Version used: 3.0.2
* Configuration: Above
* Environment name and version (e.g. Kubernetes? What version?): Docker with `/var/log` and `/run/log` mounted as volumes.
* Operating System and version: Debian 12, Redhat 8.
* Filters and plugins: systemd

**Additional context**
I deploy on various systems, some have the logs in `run` and some in `var`, so I cannot use `Path`.

In https://github.com/fluent/fluent-bit/blob/master/plugins/in_systemd/systemd_config.c#L103, `sd_journal_open(&ctx->j, SD_JOURNAL_LOCAL_ONLY);` is used to open the journal. I ran strace and found that it *does* open `/var/log/journal` and gets a file descriptor for it, but it doesn't search inside it...

I tried to configure 2 entries with `Path` - one for each path, but since each of them doesn't always exist, fluent-bit failed to start.

[patrick-stephens]

Not sure I follow the failing to start with two separate path entries, I'd expect it just to report it can't read one - same as if it did not have access for example. Unless it's to do with nesting or something strange.

Are you using two different tail inputs or specifying two paths in a single input?

[orgads]

I tried this:

[INPUT]
    Name              systemd
    Tag               systemd
    Systemd_Filter    _COMM=systemd
    Path              /run/log/journal
    Lowercase         On
    Strip_Underscores On
    DB                /fluent-bit/db/cursor1.db

[INPUT]
    Name              systemd
    Tag               systemd
    Systemd_Filter    _COMM=systemd
    Path              /var/log/journal
    Lowercase         On
    Strip_Underscores On
    DB                /fluent-bit/db/cursor2.db

and it failed with this error:

[2024/04/16 16:45:11] [ info] [input:systemd:systemd.0] initializing
[2024/04/16 16:45:11] [ info] [input:systemd:systemd.0] storage_strategy='memory' (memory only)
[2024/04/16 16:45:11] [error] [/src/fluent-bit/plugins/in_systemd/systemd_config.c:81 errno=2] No such file or directory
[2024/04/16 16:45:11] [error] [input:systemd:systemd.0] given path /run/log/journal is invalid
[2024/04/16 16:45:11] [error] [input:systemd:systemd.0] cannot initialize
[2024/04/16 16:45:11] [error] failed initialize input systemd.0
[2024/04/16 16:45:11] [error] [engine] input initialization failed

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[orgads]

@patrick-stephens ping