search

containerd / nerdctl

contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...
Apache License 2.0
7.92k stars 589 forks source link

Logging drivers enhancement #1039

Open fahedouch opened 2 years ago

fahedouch commented 2 years ago

What is the problem you're trying to solve

Target Nerdctl logging drivers:

Describe the solution you'd like

Configure logging drivers

Additional context

No response

Zheaoli commented 2 years ago

Maybe I can handle the rest of the task in this issue. @fahedouch would you mind this issue to me?

fahedouch commented 2 years ago

thank you for you request @Zheaoli . I start writing code for journald so let me finish it based on your refacto https://github.com/containerd/nerdctl/pull/1050 . You can enhance the rest of the list

Zheaoli commented 2 years ago

@fahedouch Great! Maybe I can write the fluentd and syslog first. BTW, maybe we need spilt the logger in two parts: the readable logger and the unreadable logger

fahedouch commented 2 years ago

@Zheaoli What do you mean by readable logger and the unreadable logger here ?

Zheaoli commented 2 years ago

Yes, because some of the loggers such as fluentd can not read the log directly after we wrote the log. So I think maybe we need to split the logger by two types

AkihiroSuda commented 2 years ago

Probably we should support using multiple drivers simultaneously. Then we can read/write jsonfiles while writing to fluentd.

Zheaoli commented 2 years ago

Probably we should support using multiple drivers simultaneously. Then we can read/write jsonfiles while writing to fluentd.

SGTM.

fahedouch commented 2 years ago

Probably we should support using multiple drivers simultaneously. Then we can read/write jsonfiles while writing to fluentd.

We are creating a dependency between drivers here; json-file driver issue will impact fluentd driver from nerdctl point of view. This can be voluntary when the user configures generated jsonfiles as fluentd tail input (https://docs.fluentd.org/input/tail)

I think we should implement fluentd driver as a separated LoggerFunc, we can implement one of the fluentd input using this module https://github.com/fluent/fluent-logger-golang

fahedouch commented 2 years ago

@Zheaoli FYI I start working on local driver.

Zheaoli commented 2 years ago

I have a question, should we make the log driver 100% compatible with Docker? Some of the drivers are easy to implement but hard to E2E tests, such as AWS Cloudwatch log driver or others. @fahedouch @AkihiroSuda

fahedouch commented 2 years ago

@Zheaoli implement the above list is enough at least for the moment

Zheaoli commented 2 years ago

@Zheaoli implement the above list is enough at least for the moment

SGTM

AkihiroSuda commented 2 years ago

I think the AWS CloudWatch driver should be implemented as a third party plugin that can be called with --log-driver binary://foo

Zheaoli commented 2 years ago

I think the AWS CloudWatch driver should be implemented as a third party plugin that can be called with --log-driver binary://foo

SGTM, I think I can make a design proposal

manugupt1 commented 2 years ago

I would like to try working on fluentbit driver if that is fine and no one is working on it.

AkihiroSuda commented 2 years ago

@containerd/nerdctl-maintainers

Are we fine to add the fluentbit driver as a built-in driver, or should it be an external driver (--log-driver binary://foo) ?

fahedouch commented 2 years ago

@AkihiroSuda I would say external driver; fluentbit driver(s) are be built as go plugin https://docs.fluentbit.io/manual/development/golang-output-plugins#build-a-go-plugin which can be easily handled as external driver. Also fluentbit have multiple deps so I think it is better to isolate it as external plugin.

@manugupt1 Yes you can start working on this. I started implementing the local driver which also an external plugin. We just have to agree on the structuring of external plugins

manugupt1 commented 2 years ago

@fahedouch do you have a draft PR for local that I can take inspiration from for external driver?

fahedouch commented 2 years ago

@manugupt1 Sure let me cleanup what I did and push it ASAP

Zheaoli commented 2 years ago

@manugupt1 is there any update about this issue?

manugupt1 commented 2 years ago

@manugupt1 is there any update about this issue?

Sorry, been busy with work a lot. So, I have not made much progress

manugupt1 commented 2 years ago

@AkihiroSuda @fahedouch @Zheaoli Should we close this as third party support is open or keep it open for inbuilt drivers?

dalvarezquiroga commented 1 year ago

Yeah would be great to read the logs using directly nerdctl :-)

nerdctl -n k8s.io container logs e780b0e8aba7 WARN[0000] Currently, nerdctl logs only supports containers created with nerdctl run -d FATAL[0000] open /var/lib/nerdctl/1935db59/containers/e780b0e8aba78200dfe6112dc834925952ee6bb3a39f64cb8ab88081c77c74f9/log-config.json: no such file or directory

anonymousr007 commented 1 year ago

Hello Everyone,

I am Rishabh, an undergraduate computer engineering student. I am interested in this issue. Is it still open?

Best Regards Rishabh Singh

fahedouch commented 1 year ago

Hello Everyone,

I am Rishabh, an undergraduate computer engineering student. I am interested in this issue. Is it still open?

Best Regards Rishabh Singh

Hi Rishabh, Yes, please go ahead. What lnterest you in this ticket ?

anonymousr007 commented 1 year ago

Hello Everyone, I am Rishabh, an undergraduate computer engineering student. I am interested in this issue. Is it still open? Best Regards Rishabh Singh

Hi Rishabh, Yes, please go ahead. What lnterest you in this ticket ?

I guess, These three logging drivers are not implemented yet.

austinvazquez commented 10 months ago

Hi @fahedouch, can this ticket also include adding support for "none" logging driver?

The use case is to be able to disable container logs from compose file.

e.g. nerdctl compose up of docker-compose.yml

version: "3.7"

services:
  compute:
    image: public.ecr.aws/ubuntu/ubuntu:22.04
    logging:
      driver: "none"

results in:

INFO[0000] Ensuring image public.ecr.aws/ubuntu/ubuntu:22.04
INFO[0000] Re-creating container repro-compute-1
FATA[0000] unknown logging driver "none": not found
FATA[0000] error while creating container repro-compute-1: exit status 1

References: [1] https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers

vsiravar commented 9 months ago

@fahedouch are you working on the local log driver? Do you have a draft for the design for inspiration :). I can help with the implementation.

fahedouch commented 9 months ago

@vsiravar unfortunately I didn't have enough time to make a clear enough design. The idea is to separate the local driver into a different binary then consume it using containerd runtime v2 logging plugin mode.

"binary://BIN?KEY=VALUE" URI is parsed into Args {BIN, KEY, VALUE}.