Unable to retrieve journald logs, invalid boot config

[miyunari]

Component(s)

receiver/journald

What happened?

Description

In a first step I would like to read firewalld logs and print them via the collector to my console.

Steps to Reproduce

I downloaded the opentelemetry collector contrib binary.

curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.86.0/otelcol-contrib_0.86.0_linux_amd64.tar.gz\ntar -xvf otelcol_0.86.0_linux_amd64.tar.gz\n
tar -xvf otelcol-contrib_0.86.0_linux_amd64.tar.gz\n

Then I started it with my configuration:

./otel-contrib --config=test.yaml

Expected Result

See my journald logs in the opentelemetry collector logs.

Actual Result

Getting an error message, see log output.

Collector version

v0.86.0

Environment information

Environment

OS: Fedora 38

OpenTelemetry Collector configuration

receivers:
  journald:
    directory: /run/log/journal
    units:
      - firewalld
    priority: info

processors:
  batch:

exporters:
  debug:

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    logs:
      receivers: [journald]
      processors: [batch]
      exporters: [debug]

Log output

Error: cannot start pipelines: start stanza: journalctl command failed (exit status 1): Data from the specified boot (+0) is not available: No such boot ID in journal

2023/10/11 15:17:13 collector server run finished with error: cannot start pipelines: start stanza: journalctl command failed (exit status 1): Data from the specified boot (+0) is not available: No such boot ID in journal

Additional context

I checked that the unit is active

$ systemctl list-unit-files --all
UNIT FILE                                                                           STATE           PRESET 
firewalld.service                                                                   enabled         enabled 

Using journalctl -u firewalld I also see some logs:

...
Nov 01 13:30:50 fedora systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon.
-- Boot dcede6e547a14d32bdae28ac8c2a8f3b --
Nov 01 13:39:38 fedora systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...
Nov 01 13:39:38 fedora systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
Nov 01 13:40:24 fedora systemd[1]: Stopping firewalld.service - firewalld - dynamic firewall daemon...
Nov 01 13:40:24 fedora systemd[1]: firewalld.service: Deactivated successfully.
Nov 01 13:40:24 fedora systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon.
-- Boot f9f3d62b4ca74c61b148974d1a8ef361 --

[github-actions[bot]]

Pinging code owners:

• receiver/journald: @sumo-drosiek @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sumo-drosiek]

This errors is fired up, because probably there is no files in the specified directory.

I get the same if I run the command manually

➜  opentelemetry-collector-contrib git:(main) ✗ journalctl --utc --output=json --follow --unit firewalld --priority info --directory /run/log/journal
No journal boot entry found from the specified boot offset (+0)
➜  opentelemetry-collector-contrib git:(main) ✗ ls /run/log/journal
➜  opentelemetry-collector-contrib git:(main) ✗

It works fine if directory is unspecified

[miyunari]

Thanks @sumo-drosiek! Actually the /run/log/journal folder is empty and removing the directory entry worked :smiley: But I still wonder where the journal files are located. :no_good:

2023-10-24T11:06:34.560+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 2}
2023-10-24T11:06:34.561+0200    info    ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2023-10-24 09:06:34.377994896 +0000 UTC
Timestamp: 2023-10-24 09:06:34.081978 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Map({..., "JOB_TYPE":"start","MESSAGE":"Starting firewalld.service - firewalld - dynamic firewall daemon...","MESSAGE_ID":"7d4958e842da4a758f6c1cdc7b36dcc5","PRIORITY":"6","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"systemd","TID":"1","UNIT":"firewalld.service","_BOOT_ID":"d0fae1980ca54d549daa9adfcf388ad7", ...})
Trace ID: 
Span ID: 
Flags: 0

[sumo-drosiek]

@miyunari Probably /var/log/journal. It should be somewhere in the systemd configuration :sweat_smile: