search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.72k stars 2.16k forks source link

FluentBit - Collector - Loki : Failed to transform logs into Loki log streams #16368

Closed nlamirault closed 1 year ago

nlamirault commented 1 year ago

Component(s)

exporter/loki, receiver/fluentforward

What happened?

Description

I deploy on a Kubernetes cluster FluentBit, OpenTelemetry Collector and Loki. It seems i can't send logs to Loki

Steps to Reproduce

Expected Result

Actual Result

With debug output : 

Trace ID:
Span ID:
Flags: 0
LogRecord #411
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2022-11-18 06:28:40.977852993 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str(LogRecord #10)
Attributes:
     -> fluent.tag: Str(kube.var.log.containers.logs-collector-0_opentelemetry_otc-container-f0586f930a003a23ce947ed548101312bfb3bf2b4d08f527ae239b917937b128.log)
     -> kubernetes: Map({"annotations":{"opentelemetry-operator-config/sha256":"d65f54b6cef2c021a0231f0562ed8c1d5f40945b5a16a3246d387eb4d57dc06e"},"container_hash":"docker.io/otel/opentelemetry-collector-contrib@sha256:128cf51cc7da40b68a4a
f23e788efc67c409f936d85874015f0856c911863f91","container_image":"docker.io/otel/opentelemetry-collector-contrib:0.64.0","container_name":"otc-container","docker_id":"3034545680a8559e4a4d786ab8721002500f4392c95b692cefb40b7f9c592a79","host":
"portefaix-1","labels":{"app.kubernetes.io/component":"opentelemetry-collector","app.kubernetes.io/instance":"opentelemetry.logs","app.kubernetes.io/managed-by":"opentelemetry-operator","app.kubernetes.io/name":"logs-collector","app.kubern
etes.io/part-of":"opentelemetry","app.kubernetes.io/version":"0.64.0","argocd.argoproj.io/instance":"opentelemetry-collector","controller-revision-hash":"logs-collector-6b9d797cb6","helm.sh/chart":"opentelemetry-collector-1.0.0","statefuls
et.kubernetes.io/pod-name":"logs-collector-0"},"namespace_name":"opentelemetry","pod_id":"b424113e-c067-4e50-97d2-e3ba060579b0","pod_name":"logs-collector-0"})
     -> logtag: Str(F)
     -> stream: Str(stderr)
     -> time: Str(2022-11-18T07:28:40.977852993+01:00)

Configuration :

Collector version

0.64.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Kubernetes 1.22 Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

config:

    receivers:
      otlp:
        protocols:
          grpc:
          http:
      fluentforward:
        endpoint: 0.0.0.0:24224

    processors:

      resource:
        attributes:
          - action: insert
            key: collector.name
            value: "${KUBE_POD_NAME}"

      batch:
        send_batch_max_size: 1000
        timeout: 15s
        send_batch_size : 800

      # Data sources: traces, metrics, logs
      memory_limiter:
        limit_percentage: 90
        spike_limit_percentage: 30
        check_interval: 5s

    exporters:

      logging:
        loglevel: info

      # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md
      loki:
        endpoint: http://loki-gateway.logging.svc.cluster.local:80/loki/api/v1/push
        headers:
          X-Scope-OrgID: homelab
        tls:
          insecure: true
        labels:
          attributes:
            container_name: ""
            container_id: ""
          record:
            # Adds 'traceID' as a log label, seen as 'traceid' in Loki.
            traceID: "traceid"

    extensions:

      health_check:

      memory_ballast:
        size_in_percentage: 20

      pprof:
        endpoint: :1888

      zpages:
        endpoint: :55679

    service:

      telemetry:
        logs:
          level: info
        metrics:
          level: detailed
          address: 0.0.0.0:8888

      extensions:
        - health_check
        - memory_ballast
        - pprof
        - zpages

      pipelines:

        logs:
          receivers:
            - fluentforward
            # - otlp
          processors:
            - batch
            - memory_limiter
            - resource
            # - attributes
          exporters:
            - logging
            - loki

Log output

2022-11-18T06:41:16.744Z        info    service/telemetry.go:110        Setting up own telemetry...
2022-11-18T06:41:16.748Z        info    service/telemetry.go:140        Serving Prometheus metrics      {"address": "0.0.0.0:8888", "level": "detailed"}
2022-11-18T06:41:16.749Z        info    components/components.go:30     In development component. May change in the future.     {"kind": "exporter", "data_type": "logs", "name": "logging", "stability": "in development"}
2022-11-18T06:41:16.751Z        warn    loggingexporter@v0.64.0/factory.go:110  'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior.      {"kind": "exporter", "data_type": "logs
", "name": "logging", "loglevel": "info", "equivalent verbosity level": "normal"}
2022-11-18T06:41:16.755Z        info    lokiexporter@v0.64.0/legacy_exporter.go:59      using the legacy Loki exporter  {"kind": "exporter", "data_type": "logs", "name": "loki"}
2022-11-18T06:41:16.761Z        info    memorylimiterprocessor@v0.64.0/memorylimiter.go:143     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "total_memory_mib": 3072, "limit_percentage
": 90, "spike_limit_percentage": 30}
2022-11-18T06:41:16.761Z        info    memorylimiterprocessor@v0.64.0/memorylimiter.go:112     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 2764, "spike_limit_mib": 921,
"check_interval": 5}
2022-11-18T06:41:16.763Z        info    service/service.go:89   Starting otelcol-contrib...     {"Version": "0.64.0", "NumCPU": 4}
2022-11-18T06:41:16.763Z        info    extensions/extensions.go:41     Starting extensions...
2022-11-18T06:41:16.763Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "pprof"}
2022-11-18T06:41:16.763Z        info    pprofextension@v0.64.0/pprofextension.go:71     Starting net/http/pprof server  {"kind": "extension", "name": "pprof", "config": {"ExtensionConfig":null,"TCPAddr":{"Endpoint":":1888"},"BlockProfileFr
action":0,"MutexProfileFraction":0,"SaveToFile":""}}
2022-11-18T06:41:16.767Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "pprof"}
2022-11-18T06:41:16.767Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "zpages"}
2022-11-18T06:41:16.767Z        info    zpagesextension@v0.64.0/zpagesextension.go:64   Registered zPages span processor on tracer provider     {"kind": "extension", "name": "zpages"}
2022-11-18T06:41:16.767Z        info    zpagesextension@v0.64.0/zpagesextension.go:74   Registered Host's zPages        {"kind": "extension", "name": "zpages"}
2022-11-18T06:41:16.767Z        info    zpagesextension@v0.64.0/zpagesextension.go:86   Starting zPages extension       {"kind": "extension", "name": "zpages", "config": {"ExtensionConfig":null,"TCPAddr":{"Endpoint":":55679"}}}
2022-11-18T06:41:16.767Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "zpages"}
2022-11-18T06:41:16.767Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "health_check"}
2022-11-18T06:41:16.767Z        info    healthcheckextension@v0.64.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"ExtensionConfig":null,"Endpoint":"0.0.0.0:13133","TLS
Setting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-11-18T06:41:16.768Z        warn    internal/warning.go:51  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks        {"kind": "extension", "name": "health_check", "
documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2022-11-18T06:41:16.774Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "health_check"}
2022-11-18T06:41:16.774Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "memory_ballast"}
2022-11-18T06:41:16.863Z        info    ballastextension@v0.64.0/memory_ballast.go:55   Setting memory ballast  {"kind": "extension", "name": "memory_ballast", "MiBs": 614}
2022-11-18T06:41:16.863Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "memory_ballast"}
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:74       Starting exporters...
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "loki"}
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "logs", "name": "loki"}
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:86       Starting processors...
2022-11-18T06:41:16.863Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:98       Starting receivers...
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    pipelines/pipelines.go:106      Receiver started.       {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-11-18T06:41:16.864Z        info    healthcheck/handler.go:129      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
2022-11-18T06:41:16.864Z        info    service/service.go:106  Everything is ready. Begin running and processing data.
2022-11-18T06:41:17.987Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1000}
2022-11-18T06:41:17.990Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 829}
2022-11-18T06:41:17.992Z        error   exporterhelper/queued_retry.go:394      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "logs", "name": "loki", "error": "Permanent error: failed to
transform logs into Loki log streams", "dropped_items": 829}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/queued_retry.go:394
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
        go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/logs.go:131
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/queued_retry.go:205
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
        go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61
2022-11-18T06:41:18.995Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1000}

Additional context

No response

github-actions[bot] commented 1 year ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

jpkrohling commented 1 year ago

This is happening because there are no labels for the items that were exported. I see you had two batches, one with 1000 records and one with 829. Only the smaller batch seems to have failed. Perhaps that one didn't contain the expected log attributes?

The Loki exporter has been changed recently to export data even if no labels exist. This requires configuring the exporter in a slightly different way, like the following:

    receivers:
      otlp:
        protocols:
          grpc:
          http:
      fluentforward:
        endpoint: 0.0.0.0:24224

    processors:

      attributes:
        actions:
        - action: insert
          key: loki.attribute.labels
          value: [container_name, container_id]

      resource:
        attributes:
          - action: insert
            key: collector.name
            value: "${KUBE_POD_NAME}"

      batch:
        send_batch_max_size: 1000
        timeout: 15s
        send_batch_size : 800

      # Data sources: traces, metrics, logs
      memory_limiter:
        limit_percentage: 90
        spike_limit_percentage: 30
        check_interval: 5s

    exporters:

      logging:
        loglevel: info

      # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md
      loki:
        endpoint: http://loki-gateway.logging.svc.cluster.local:80/loki/api/v1/push
        headers:
          X-Scope-OrgID: homelab
        tls:
          insecure: true

    extensions:

      health_check:

      memory_ballast:
        size_in_percentage: 20

      pprof:
        endpoint: :1888

      zpages:
        endpoint: :55679

    service:

      telemetry:
        logs:
          level: info
        metrics:
          level: detailed
          address: 0.0.0.0:8888

      extensions:
        - health_check
        - memory_ballast
        - pprof
        - zpages

      pipelines:

        logs:
          receivers:
            - fluentforward
            # - otlp
          processors:
            - batch
            - memory_limiter
            - resource
            # - attributes
          exporters:
            - logging
            - loki

With your current configuration, you see this in the logs:

2022-11-18T06:41:16.755Z        info    lokiexporter@v0.64.0/legacy_exporter.go:59      using the legacy Loki exporter  {"kind": "exporter", "data_type": "logs", "name": "loki"}

With the new configuration, you should see something different. Note that we do NOT encourage you to use trace IDs in labels, as those are inherently high cardinality and go against the best practices for Loki. We have removed native ways of setting this value.

jpkrohling commented 1 year ago

cc @mar4uk

nlamirault commented 1 year ago

Thanks @jpkrohling i will try This configuration

nlamirault commented 1 year ago

thanks @jpkrohling it works now : 

2022-11-24T07:59:02.942Z        info    service/telemetry.go:110        Setting up own telemetry...
2022-11-24T07:59:02.943Z        info    service/telemetry.go:140        Serving Prometheus metrics      {"address": "0.0.0.0:8888", "level": "detailed"}
2022-11-24T07:59:02.943Z        info    components/components.go:30     In development component. May change in the future.     {"kind": "exporter", "data_type": "logs", "name": "logging", "stability": "in development"}
2022-11-24T07:59:02.943Z        warn    loggingexporter@v0.64.0/factory.go:110  'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior.      {"kind": "exporter", "data_type": "logs", "name": "logging", "loglevel": "info", "equivalent verbosity level": "normal"
}
2022-11-24T07:59:02.944Z        info    lokiexporter@v0.64.0/next_exporter.go:43        using the new Loki exporter     {"kind": "exporter", "data_type": "logs", "name": "loki"}
2022-11-24T07:59:02.945Z        info    memorylimiterprocessor@v0.64.0/memorylimiter.go:143     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "total_memory_mib": 3072, "limit_percentage": 90, "spike_limit_percentage": 30}
2022-11-24T07:59:02.945Z        info    memorylimiterprocessor@v0.64.0/memorylimiter.go:112     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 2764, "spike_limit_mib": 921, "check_interval": 5}
2022-11-24T07:59:02.950Z        info    service/service.go:89   Starting otelcol-contrib...     {"Version": "0.64.0", "NumCPU": 4}
2022-11-24T07:59:02.950Z        info    extensions/extensions.go:41     Starting extensions...
2022-11-24T07:59:02.950Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "health_check"}
2022-11-24T07:59:02.950Z        info    healthcheckextension@v0.64.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"ExtensionConfig":null,"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":f
alse,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-11-24T07:59:02.950Z        warn    internal/warning.go:51  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks        {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/
main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2022-11-24T07:59:02.951Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "health_check"}
2022-11-24T07:59:02.951Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "memory_ballast"}
2022-11-24T07:59:03.036Z        info    ballastextension@v0.64.0/memory_ballast.go:55   Setting memory ballast  {"kind": "extension", "name": "memory_ballast", "MiBs": 614}
2022-11-24T07:59:03.036Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "memory_ballast"}
2022-11-24T07:59:03.036Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "pprof"}
2022-11-24T07:59:03.037Z        info    pprofextension@v0.64.0/pprofextension.go:71     Starting net/http/pprof server  {"kind": "extension", "name": "pprof", "config": {"ExtensionConfig":null,"TCPAddr":{"Endpoint":":1888"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2022-11-24T07:59:03.037Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "pprof"}
2022-11-24T07:59:03.037Z        info    extensions/extensions.go:44     Extension is starting...        {"kind": "extension", "name": "zpages"}
2022-11-24T07:59:03.037Z        info    zpagesextension@v0.64.0/zpagesextension.go:64   Registered zPages span processor on tracer provider     {"kind": "extension", "name": "zpages"}
2022-11-24T07:59:03.037Z        info    zpagesextension@v0.64.0/zpagesextension.go:74   Registered Host's zPages        {"kind": "extension", "name": "zpages"}
2022-11-24T07:59:03.037Z        info    zpagesextension@v0.64.0/zpagesextension.go:86   Starting zPages extension       {"kind": "extension", "name": "zpages", "config": {"ExtensionConfig":null,"TCPAddr":{"Endpoint":":55679"}}}
2022-11-24T07:59:03.037Z        info    extensions/extensions.go:48     Extension started.      {"kind": "extension", "name": "zpages"}
2022-11-24T07:59:03.037Z        info    pipelines/pipelines.go:74       Starting exporters...
2022-11-24T07:59:03.037Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-11-24T07:59:03.037Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-11-24T07:59:03.037Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "loki"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "logs", "name": "loki"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:86       Starting processors...
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:98       Starting receivers...
2022-11-24T07:59:03.038Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-11-24T07:59:03.039Z        info    pipelines/pipelines.go:106      Receiver started.       {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-11-24T07:59:03.039Z        info    healthcheck/handler.go:129      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
2022-11-24T07:59:03.039Z        info    service/service.go:106  Everything is ready. Begin running and processing data.
2022-11-24T07:59:18.043Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 614}
2022-11-24T07:59:33.047Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 511}
2022-11-24T07:59:38.439Z        info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "loki", "error": "HTTP 500 \"Internal Server Error\": rpc error: code = DeadlineExceeded desc = context deadline exceeded", "
interval": "3.552446007s"}
2022-11-24T07:59:48.049Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 45}
2022-11-24T08:00:03.059Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 204}
2022-11-24T08:00:18.062Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 633}
2022-11-24T08:00:29.464Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 896}
2022-11-24T08:00:44.468Z        info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 50}

I've got another problem/question : in Loki, i can choose only one filter : exporter=OTLP How can i add to filter list : namespace, pod, app, .... (aka Kubernetes labels)

2022-11-24_09-29

jpkrohling commented 1 year ago

You'd have to add a hint with the attributes to be promoted to labels:

processors:
  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: namespace_name

After that, add it to the pipeline:

  pipelines:
    logs:
      receivers: [...]
      processors: [..., attributes]
      exporters: [..., loki]

For more details on this, refer to the readme part of the Loki exporter. I'm closing this, but feel free if you think the original problem hasn't been addressed.