can not use fluentd log driver

[Hendrik-H]

/kind bug

Description According to the bash completion script in https://github.com/containers/libpod/blob/master/completions/bash/podman it seems like podman is supporting several log drivers, including fluentd. However when starting a container with the fluentd log driver fluentd is stated to be an invalid argument.

Steps to reproduce the issue:

1. podman run --name nginx --rm -it --log-driver fluentd nginx

Describe the results you received: Error: error running container create option: invalid log driver: invalid argument

Describe the results you expected: That logging via the fluentd log driver works just like it does for Docker.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.9.2
RemoteAPI Version:  1
Go Version:         go1.13.10
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.13.10
  podmanVersion: 1.9.2
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.16-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.16, commit: aec991fec16dc45935de184f2ea06a6ffca200a0'
  cpus: 12
  distribution:
    distribution: fedora
    version: "31"
  eventLogger: file
  hostname: p52
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.6.13-200.fc31.x86_64
  memFree: 10336419840
  memTotal: 33334947840
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16768823296
  swapTotal: 16768823296
  uptime: 46h 2m 29.96s (Approximately 1.92 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 2
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes

[mheon]

Looks like a bug in the completions script; we do not presently support the fluentd log driver.

On Mon, May 25, 2020, 11:30 Hendrik-H notifications@github.com wrote:

/kind bug

Description According to the bash completion script in https://github.com/containers/libpod/blob/master/completions/bash/podman it seems like podman is supporting several log drivers, including fluentd. However when starting a container with the fluentd log driver fluentd is stated to be an invalid argument.

Steps to reproduce the issue:

1. podman run --name nginx --rm -it --log-driver fluentd nginx

Describe the results you received: Error: error running container create option: invalid log driver: invalid argument

Describe the results you expected: That logging via the fluentd log driver works just like it does for Docker.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version: 1.9.2 RemoteAPI Version: 1 Go Version: go1.13.10 OS/Arch: linux/amd64

Output of podman info --debug:

debug: compiler: gc gitCommit: "" goVersion: go1.13.10 podmanVersion: 1.9.2 host: arch: amd64 buildahVersion: 1.14.8 cgroupVersion: v2 conmon: package: conmon-2.0.16-2.fc31.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.16, commit: aec991fec16dc45935de184f2ea06a6ffca200a0' cpus: 12 distribution: distribution: fedora version: "31" eventLogger: file hostname: p52 idMappings: gidmap: null uidmap: null kernel: 5.6.13-200.fc31.x86_64 memFree: 10336419840 memTotal: 33334947840 ociRuntime: name: crun package: crun-0.13-2.fc31.x86_64 path: /usr/bin/crun version: |- crun version 0.13 commit: e79e4de4ac16da0ce48777afb72c6241de870525 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL os: linux rootless: false slirp4netns: executable: "" package: "" version: "" swapFree: 16768823296 swapTotal: 16768823296 uptime: 46h 2m 29.96s (Approximately 1.92 days) registries: search:

• registry.fedoraproject.org
• registry.access.redhat.com
• registry.centos.org
• docker.io store: configFile: /etc/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev,metacopy=on graphRoot: /var/lib/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "true" imageStore: number: 2 runRoot: /var/run/containers/storage volumePath: /var/lib/containers/storage/volumes

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/6377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3AOCFSPCNKODOYSS53CXDRTKFILANCNFSM4NJTQEFQ .

[Hendrik-H]

What log drivers are supported? There seems to be some support for journald but entries are logged as binary and thus journalctl -a is required to see them and the CONTAINER_NAME field does not seem to be set.

Is the support for logging drivers planned anytime soon?

[mheon]

We presently support journald and k8s-file (which is somewhat similar to Docker's json-file but presented in a standard format used by Kubernetes and the CRI). Support for CONTAINER_NAME is coming soon (probably Podman v2.0; last I checked, all necessary changes for it have merged).

Proper logging drivers are more tricky, because Podman itself isn't doing the logging, but conmon (our very minimal container monitor program). I think we could trivially add support for the syslog and none log drivers, but anything more complex than that will require significant changes to Conmon and I'm not sure when the team will have time to make those. If anyone in the community wants to take this up, we'd be happy to accept patches!

[Hendrik-H]

I tried the k8s-file log driver now using podman run --name nginx --rm -it --log-driver k8s-file --log-opt path=/root/mycontainer.json nginx but that resulting log file does not contain JSON but what I see as stdout prepanded with a timestamp and "stdout". Lines look like this:

2020-05-25T18:57:44.716251964+02:00 stdout F 10.88.0.1 - - [25/May/2020:16:57:44 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" "-"

The call looks just like what is documented in http://docs.podman.io/en/latest/markdown/podman-create.1.html, which actually also lists the supported log drivers :-)

[rhatdan]

@haircommander PTAL I think implementing the syslog and none drivers is a good idea.

[haircommander]

@Hendrik-H as I understand it, fluentd can also consume the cri format logs (which k8s-file uses). does that satisfy your needs? I opened https://github.com/containers/conmon/issues/159 to track the drivers.

[Hendrik-H]

@haircommander I couldn't find any documentation on fluentd supporting a cri format. I found this however, which seems to indicate that it is not supported: https://github.com/fluent/fluent-bit/issues/522

Using k8s-file I would get a log file that grows over time, which can cause problems. So I would need to use something like logrotate to get around that. But then I would have to hope that fluentd doesn't get a problem with that rotation and also doesn't miss any entries due to a rotation. Another option I looked at was to pipe the output of podman into fluentd and then use a parser to create log entries. This has the problem that I can not save the stdout vs stderr information and also don't get a timestamp.

So currently I'm looking into wrapping the podman call and then reading stdout and stderr and sending it on my own to fluentd. If podman wants to be a full replacement of docker it would however be a lot nicer if it would provide the same log driver support or something that allows others to easily plug something in ;-)

[haircommander]

https://github.com/fluent/fluent-bit/pull/667 seems to indicate to me that issue was closed because it was fixed, but I have done no work to verify that :smile:

understood adding the support would be best though. AFAICT the main maintainers (myself included) are pretty swamped and would happily accept PRs that would cover this gap in features :slightly_smiling_face:

[Hendrik-H]

Hm, right, odd. I actually wanted to link to https://github.com/fluent/fluent-bit/issues/522 But looks like they indeed added regex parser rule for CRI. Too many open windows ;-) Parsing what was just written is however also not really ideal. My Go skill is unfortunately not quite on the right level to submit a PR anytime soon.

[haircommander]

I am not sure what your C skills look like, but conmon is written in C :smiley: I'd be happy to guide you on how the log driver would be implemented

[mheon]

Going to relabel as kind/feature as this is technically new functionality

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Hendrik-H Did you get a chance to look at this?

@ParkerVR or @ryanchpowell Want to take a look at this?

[Hendrik-H]

Sorry, I did not look into the implementation. It is quite unlikely that I will be able to provide a fix any time soon. We will likely grab the stout/stderr output and than process it accordingly.

On July 6, 2020 6:49:53 PM UTC, Daniel J Walsh notifications@github.com wrote:

@Hendrik-H Did you get a chance to look at this?

@ParkerVR or @ryanchpowell Want to take a look at this?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/containers/libpod/issues/6377#issuecomment-654406173

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

Would the journald log driver work?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we have had no response on this for a couple of months, closing.

[MrWong99]

We are currently considering implementing the FluentD driver to be able to use it with podman and podman-compose. As we are fairly new to the podman source code: could you possibly point us to the packages that would likely need changes? From what I can gather:

• libpod/container_log.go -> From what I can understand the logs should also be readable back from FluentD then? This is not exactly what Docker does for the driver from what I see. I think Docker only sends the logs to FluentD.
• add e2e tests
• I am missing where the logs are actually saved/synced from the containers. Is this implemented somewhere else and not this project?

[rhatdan]

@mheon could you answer the above question^^

[mheon]

The core changes required will not be in Podman or Libpod, but in Conmon, our ultra-minimal monitor process for containers. There, you'll need to add support for writing fluentd logs. Once that is done, you will need to modify container_log.go as you noted to support reading logs from fluentd, and make a scattering of small changes elsewhere in the project (probably mostly in Libpod and cmd/podman) to allow fluentd as valid input to the --log-driver flag. After that, E2E tests and you're good.

[MrWong99]

Thanks @mheon we will have a look :)

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I think this is still waiting for conmon-rs to become a reality.

[jesselang]

Perhaps in the meantime, fluentd should be removed from the completion script? I can take a swing at a PR for that.

[jesselang]

Oops, never mind. Looks like fluentd is no longer an option in the completion script.

[Neopixler]

Hi, I would like to switch a container from Docker to Podman. The container uses the fluentd log-driver. Is it planned to provide it in future?

[MrWong99]

After I said this:

We are currently considering implementing the FluentD driver to be able to use it with podman and podman-compose. As we are fairly new to the podman source code: could you possibly point us to the packages that would likely need changes?

I took a deeper look into the matter and realized that there would be many changes needed. As I understand it, Podman currently expects log-drivers to be read and write which Docker doesn't do, since log-drivers like fluentd are just forwarding logs and have no options to read back from them.

But I am no maintainer of this project so I can't give you any answers to this question @Neopixler

[Neopixler]

No issue, I discussed it with one of my colleague and we will use the journald log-driver instead.

[rhatdan]

If the log driver is read/only what happens when users execute docker logs?

[MrWong99]

When we used the fluentd-driver (this was over a year ago) if you'd run "docker logs" the output would just be empty.

[rhatdan]

Ok I think we could do something similar for Podman.

[yonatanbsepio]

Hi, does anyone know if the fluentd-driver source code is proprietary Docker code, or if it is available somewhere?

I'd want to take a look to gauge the complexity of getting into implementing the driver for Podman

[MrWong99]

I found this in the Docker code: https://github.com/moby/moby/blob/master/daemon/logger/fluentd/fluentd.go

I am no maintainer, so no guarantees!