k8sobjects to elasticsearch failed for the field app

[MaheshSankaran]

Component(s)

receiver/k8sobjects

What happened?

Description

Trying to configure k8s object events to elasticsearch

Steps to Reproduce

receivers:
      k8sobjects:
        auth_type: serviceAccount
        objects:
          - name: events
            mode: watch
          - name: pods
            mode: pull

    exporters:
      logging:
        verbosity: detailed
      elasticsearch/events:
        endpoints: [https://elasticsearch-es-internal-http:9200]
        index: indexname
        user: elastic
        password: elastic
        sending_queue:
          enabled: true
          num_consumers: 1
          queue_size: 1000
        tls:
          insecure_skip_verify: true

    service:
      pipelines:
        logs:
          receivers:
          - k8sobjects
          exporters:
          - logging
          - elasticsearch/events

Expected Result

Actual Result

error elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:309 Drop docs: failed to index: struct { Type string "json:\"type\""; Reason string "json:\"reason\"" }{Type:"document_parsing_exception", Reason:"[1:447] object mapping for [Body.metadata.labels.app] tried to parse field [app] as object, but found a concrete value"} {"kind": "exporter", "data_type": "logs", "name": "elasticsearch/events", "status": 400} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(worker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:309 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(worker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:291 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:207

Collector version

v0.103.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
      k8sobjects:
        auth_type: serviceAccount
        objects:
          - name: events
            mode: watch
          - name: pods
            mode: pull

    exporters:
      logging:
        verbosity: detailed
      elasticsearch/events:
        endpoints: [https://elasticsearch-es-internal-http:9200]
        index: indexname
        user: elastic
        password: elastic
        sending_queue:
          enabled: true
          num_consumers: 1
          queue_size: 1000
        tls:
          insecure_skip_verify: true

    service:
      pipelines:
        logs:
          receivers:
          - k8sobjects
          exporters:
          - logging
          - elasticsearch/events

Log output

error   elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:309    Drop docs: failed to index: struct { Type string "json:\"type\""; Reason string "json:\"reason\"" }{Type:"document_parsing_exception", Reason:"[1:447] object mapping for [Body.metadata.labels.app] tried to parse field [app] as object, but found a concrete value"} {"kind": "exporter", "data_type": "logs", "name": "elasticsearch/events", "status": 400}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*worker).flush
    github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:309
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*worker).run
    github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:291
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
    github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.103.0/elasticsearch_bulk.go:207

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• receiver/k8sobjects: @dmitryax @hvaghani221 @TylerHelmuth

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

Pinging code owners for exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

This seems like a problem with elastic search translation not the k8s object receiver.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

@MaheshSankaran hi, Can you desensitize the event data and put it here? So we can do a better analysis