fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.74k stars 1.56k forks source link

Forwarding linux journal to stack driver produced illegal log name character errors #9254

Open jum opened 4 weeks ago

jum commented 4 weeks ago

Bug Report

When forwarding the full journal of a linux host to stack driver, the character '@' in some of the systemd unit names produced errors with stack driver.

The '@' signs in log names for stack driver should be transcribed to the valid char set.

[2024/06/21 22:53:51] [ warn] [output:stackdriver:stackdriver.0] tag=host.postf>
  "error": {
    "code": 400,
    "message": "Log name (host.postfix@-.service) contains illegal character. A>
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.logging.v2.WriteLogEntriesPartialE>
        "logEntryErrors": {
          "1": {
            "code": 3,
            "message": "Log name (host.postfix@-.service) contains illegal char>
          },
          "0": {
            "code": 3,
            "message": "Log name (host.postfix@-.service) contains illegal char>
          }
        }
      }
    ]
  }
}

Just forward all the logs to google stack driver.

Expected behavior

Should log all units, even those with a '@' sign in their name.

Your Environment

I use the following config:

[INPUT]
    name systemd
    tag host.*
    DB /var/lib/fluent-bit/journal.db

[OUTPUT]
    name  stackdriver
    match *
    google_service_credentials /path/to/credentials.json
    resource gce_instance
    resource_labels instance_id=worker0,zone=hosteurope
apt info fluent-bit
Package: fluent-bit
Version: 3.1.6
Priority: optional
Section: devel
Maintainer: Eduardo Silva <eduardo@calyptia.com>
Installed-Size: 128 MB
Depends: libc6 (>= 2.29), libgcc-s1 (>= 4.2), libpq5 (>= 9.0~), libsasl2-2 (>= 2.1.27+dfsg), libssl1.1 (>= 1.1.1), libsystemd0 (>= 221), libyaml-0-2, zlib1g (>= 1:1.2.0)
Download-Size: 42.3 MB
APT-Manual-Installed: yes
APT-Sources: https://packages.fluentbit.io/debian/bullseye bullseye/main amd64 Packages

Additional context

I am unable to get most of the logs relating to postfix and other systemd units that use '@' in their unit names into my central log repository in google cloud stack driver logging.

Just changing the '@' sign to something like 'AT' would help a lot as one would be able to see these log entries in the central log and filter on them.

nandra commented 1 week ago

Same problem here with version 1.9.9