vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.67k stars 1.56k forks source link

journald current_boot_only = false no longer works with systemd >= 250 #18068

Open mmelvin0 opened 1 year ago

mmelvin0 commented 1 year ago

A note for the community

Problem

As of systemd v250, journalctl --follow now implies --boot, which selects only journal entries from the current boot. This means that current_boot_only = false in Vector no longer works. It's not possible (without a workaround) to batch-load more than a single boot worth of journal or to ensure entries logged after Vector has exited during a reboot get logged after the systems comes back up.

Configuration

[sources.journal]
type = 'journald'
current_boot_only = false

Version

vector 0.31.0 (x86_64-unknown-linux-gnu)

Debug Output

No response

Example Data

No response

Additional Context

Here's a hacky (and mostly un-tested) work-around since I can't currently come up with an elegant solution to fix either the Vector or systemd side. It's a wrapper you can set as your journalctl_path:

#!/bin/bash

export SYSTEMD_COLORS=false
export SYSTEMD_PAGER=

args=()

for arg in "$@"; do
  if [[ "$arg" =~ ^--after-cursor=(.*)$ ]]; then
    cursor="${BASH_REMATCH[1]}"
  elif [[ "$arg" =~ ^--since=(.*)$ ]]; then
    since="${BASH_REMATCH[1]}"
  elif [[ "$arg" == '--boot' ]]; then
    boot='current'
  fi
  args+=("$arg")
done

if [[ "$since" != 'now' && "$boot" != 'current' ]]; then
  end="$(journalctl -n1 -b-1 -ojson | jq -r '.__REALTIME_TIMESTAMP | tonumber / 1000000')"
  if [[ $? == 0 && "$end" != '' ]]; then
    if [[ "$cursor" != '' ]]; then
      start="--after-cursor=$cursor"
    else
      start="-S$since"
    fi
    journalctl -a -ojson "$start" -U"@$end"
  fi
fi

exec journalctl "${args[@]}"

References

The code in journalctl has evolved a little bit since this change, but this is the relevant change in systemd: https://github.com/systemd/systemd/commit/2dd9285bacc2212e20853312f44175303c84807b#diff-02b215a979685f0dfd90eb90842bbd2cb408f63e87f70d4e7805a514402086caR562-R566

neuronull commented 1 year ago

While I did not test this locally, I do see the referenced systemd change (thank you) , and in the code for the vector source, we are setting --follow. This confirms that when current_boot_only = false, is configured in vector, it no longer has an effect.