search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.12k stars 2.39k forks source link

opentelementry receiver log from file and export to opensearch #34566

Open NamanGajjar26 opened 3 months ago

NamanGajjar26 commented 3 months ago

Component(s)

exporter/opensearch

What happened?

Description

hi there i am try to send receivers log file from this path (/app/log/naman/namanlog.log) and export this log to opensearch and make new index (demo1-index) get this log opensearch

Steps to Reproduce

     ` config:
        receivers:    
          filelog:
            include: ["/app/log/naman/namanlog.log"] 
            start_at: beginning      

        processors:
          batch:
            timeout: 60s
            send_batch_max_size: 10000

        extensions:
          basicauth/client:
            client_auth:
              username: admin
              password: TadhakDev01
          health_check:
            endpoint: ${env:MY_POD_IP}:13133

        exporters: 
          logging:
            loglevel: debug  
          opensearch:
            logs_index: otel-index-opensearch
            http:
              endpoint: http://opensearch-cluster-master-headless.myopensearch.svc.cluster.local:9200
              auth:
                authenticator: basicauth/client
              tls:
               insecure: true

        service:    
          extensions:
            - basicauth/client
            - health_check
          pipelines:           
            logs:
              receivers: [filelog]        
              exporters: [opensearch]`

Expected Result

i want to get this log message in opensearch using opentelemetry {"@timestamp":"2024-08-09T13:33:12.406Z","flb-key":"kube.custommvc.app.log.naman.namanlog.log","Timestamp":"2024-08-09T13:33:12.1972500+00:00","Level":"Error","MessageTemplate":"time_error_new2","TraceId":"b6475c2b94d380210de53197e5c482e8","SpanId":"53a55dc04afce4b5","Properties":{"Method":"errorlog","IndiaTime":"2024-08-09T19:03:12.1972334","UtcTimestamp":"2024-08-09T13:33:12.1972475Z","Timestamp":"2024-08-09T13:33:12.1972429+00:00"}}

Actual Result

there are show error in opentelemetry error error reader/reader.go:140 process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/app/log/naman/namanlog.log", "error": "failed to process the docker log: move: field log does not exist"}

Screenshot from 2024-08-09 19-16-29

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:    
              filelog:
                include: ["/app/log/naman/namanlog.log"] 
                start_at: beginning      

            processors:
              batch:
                timeout: 60s
                send_batch_max_size: 10000

            extensions:
              basicauth/client:
                client_auth:
                  username: admin
                  password: TadhakDev01
              health_check:
                endpoint: ${env:MY_POD_IP}:13133

            exporters: 
              logging:
                loglevel: debug  
              opensearch:
                logs_index: otel-index-opensearch
                http:
                  endpoint: http://opensearch-cluster-master-headless.myopensearch.svc.cluster.local:9200
                  auth:
                    authenticator: basicauth/client
                  tls:
                   insecure: true          
            service:    
              extensions:
                - basicauth/client
                - health_check
              pipelines:           
                logs:
                  receivers: [filelog]        
                  exporters: [opensearch]

Log output

github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/internal/reader/reader.go:140
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/file.go:160
2024-08-09T13:36:29.945Z error reader/reader.go:140 process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/app/log/naman/namanlog.log", "error": "failed to process the docker log: move: field log does not exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/internal/reader/reader.go:140
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/file.go:160
2024-08-09T13:36:29.945Z error reader/reader.go:140 process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/app/log/naman/namanlog.log", "error": "failed to process the docker log: move: field log does not exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/internal/reader/reader.go:140
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/file.go:160
2024-08-09T13:36:29.945Z error reader/reader.go:140 process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/app/log/naman/namanlog.log", "error": "failed to process the docker log: move: field log does not exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/internal/reader/reader.go:140
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/file.go:160
2024-08-09T13:36:29.945Z error reader/reader.go:140 process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/app/log/naman/namanlog.log", "error": "failed to process the docker log: move: field log does not exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/internal/reader/reader.go:140
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.106.1/fileconsumer/file.go:160

Additional context

No response

github-actions[bot] commented 3 months ago

Pinging code owners:

NamanGajjar26 commented 3 months ago

i am get this two error below how to solved this error

error opensearchexporter@v0.106.1/logger.go:36 Request failed. {"kind": "exporter", "data_type": "logs", "name": "opensearch", "path": "/_bulk", "method": "POST", "duration": 0.001234298, "reason": "net/http: HTTP/1.x transport connection broken: http: ContentLength=22616 with Body length 0"}

error exporterhelper/common.go:296 Exporting failed. Rejecting data. {"kind": "exporter", "data_type": "logs", "name": "opensearch", "error": "not retryable error: Permanent error: Permanent error: flush: net/http: HTTP/1.x transport connection broken: http: ContentLength=22616 with Body length 0\nPermanent error: Permanent error: flush: net/http: HTTP/1.x transport connection broken: http: ContentLength=22616 with Body length 0", "rejected_items": 12}

and i am config in yml like below

  `config:
    receivers:    
      filelog:
        include: ["/app/log/naman/namanlog.log"] 
        start_at: end
        operators:
          - type: regex_parser
            regex: '"Timestamp":"(?P<Timestamp>[^"]+)","Level":"(?P<Level>[^"]+)","MessageTemplate":"(?P<MessageTemplate>[^"]+)","TraceId":"(?P<TraceId>[^"]+)","SpanId":"(?P<SpanId>[^"]+)","Properties":{(?P<Properties>.*)}'

    processors:
      batch:
        timeout: 60s
        send_batch_max_size: 10000

    extensions:
      basicauth/client:
        client_auth:
          username: admin
          password: TadhakDev01
      health_check:
        endpoint: ${env:MY_POD_IP}:13133

    exporters: 
      logging:
        loglevel: debug  
      opensearch:
        logs_index: otel-index-opensearch
        http:
          endpoint: http://opensearch-cluster-master-headless.myopensearch.svc.cluster.local:9200
          auth:
            authenticator: basicauth/client
          tls:
           insecure: true
           insecure_skip_verify: true 
          timeout: 60s

    service:
      telemetry:
        logs:
          level: debug    
      extensions:
        - basicauth/client
        - health_check
      pipelines:           
        logs:
          receivers: [filelog]        
          exporters: [opensearch]`