Alloy running in a container with loki.source.journal unable to scrape journald logs from host if path is not specified

suever commented 1 month ago

What's wrong?

According to the documentation for the loki.source.journal component, if no path is specified for the loki.source.journal component, then /var/log/journal is used by default .

When using the very basic configuration from the example:

loki.source.journal "LABEL" {
  forward_to    = RECEIVER_LIST
}

When running alloy with the loki.source.journal component configured as above on a host directly, logs are discovered and scraped properly; however, when running within a container on the host and mounting /var/log/journal as a volume into the container, the logs are not discovered.

After digging into the code, this is due to the fact that, when no path is specified, alloy takes a different code path which uses go-systemd's defaults for discovering the journal entries which uses the SD_JOURNAL_LOCAL_ONLY option which ensures that only logs generated on the local machine are discovered. If you explicitly specify a path configuration option to the loki.source.journal component, then journal entries are discovered a different way which does not specify the SD_JOURNAL_LOCAL_ONLY option, which does not limit it to local logs only and it is able to discover the logs from the host /var/log/journal.

I'm not sure if this is a documentation issue for the component where it should state that you must set path = "/var/log/journal" when running in Docker, or probably more appropriately a code change where the default path is specified explicitly in alloy codebase and then go-systemd/sdjournal is invoked the same way regardless of whether a path was supplied or not which removes the behavior of alloy being dependent upon go-systemd's implementation.

I am happy to contribute a fix, but wanted to let the maintainers determine what is most appropriate to avoid confusion for users like this one

Steps to reproduce

Create the following config in config.alloy

loki.source.journal "read" {
  forward_to    = [ loki.write.loki.receiver ]
  labels        = {component = "loki.source.journal"}
}

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway:8080/loki/api/v1/push"
  }
}

Run it locally on a linux machine

./alloy-linux-amd64  run --server.http.listen-addr=0.0.0.0:12345 --storage.path=./alloy ./config.alloy

Ensure journal targets are discovered:

curl -s http://localhost:12345/metrics | grep 'loki_source_journal_target_lines_total{'

loki_source_journal_target_lines_total{component_id="loki.source.journal.read",component_path="/"} 2580

Run it in docker:

docker run -v /var/log/journal:/var/log/journal -v `pwd`/config.alloy:/etc/alloy/config.alloy -p 12345:12345 grafana/alloy:v1.1.0 run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy

Ensure no journal targets are discovered

curl -s http://localhost:12345/metrics | grep 'loki_source_journal_target_lines_total{'

loki_source_journal_target_lines_total{component_id="loki.source.journal.read",component_path="/"} 0

Modify the config to explicitly set the path

loki.source.journal "read" {
  forward_to    = [ loki.write.loki.receiver ]
  labels        = {component = "loki.source.journal"}
  path          = "/var/log/journal"
}

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway:8080/loki/api/v1/push"
  }
}

Re-run in docker, and ensure that targets are now discovered

curl -s http://localhost:12345/metrics | grep 'loki_source_journal_target_lines_total{'

loki_source_journal_target_lines_total{component_id="loki.source.journal.read",component_path="/"} 2580

System information

Linux / Docker

Software version

v1.1.0 and master

Configuration

loki.source.journal "read" {
  forward_to    = [ loki.write.loki.receiver ]
  labels        = {component = "loki.source.journal"}
  path = "/var/log/journal"
}

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway:8080/loki/api/v1/push"
  }
}

Logs

No response

csh0101 commented 1 month ago

I think you're right.. and what can we do for this problem?

github-actions[bot] commented 5 days ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

grafana / alloy