open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.11k stars 2.39k forks source link

[exporter/syslog-exporter] Unable to send traces to Syslog sink via UDP #31130

Closed kyleprice closed 7 months ago

kyleprice commented 9 months ago

Component(s)

exporter/syslog

What happened?

Description

When receiving syslog messages and then transmitting via UDP I have the following config:

syslog: network: udp port: 1500 endpoint: 10.0.0.2 (local network connection that accepts udp only)

retry_on_failure: enabled: true

sending_queue: enabled: true

receivers: syslog: tcp: listen_address: "0.0.0.0:54500" protocol: rfc3164 location: UTC

service: pipelines: logs: exporters:

Actual Result

This produces tcp connections on the exporter instead of sending over udp. The sink I am using accepts a udp connection, as a result it fails to ship logs to it.

Exporting failed. Dropping data. Try enabling retry_on_failure config option to retry on retryable errors. {"kind": "exporter", data_type": "logs", "name": "syslog", "error": dial tcp 10.0.0.2:1500: connect: connection refused", "dropped_items": 4}

Expected Result

I would expect this to send logs over udp. I looked through the code and saw this line which looks to set the network to tcp always for the exporter. I wanted a 2nd pair of eyes to confirm this as this would match what I am seeing in my deployment.

Collector version

0.93.0

Environment information

Environment

OS: Ubuntu 20.04 Using Otel Operator

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

github-actions[bot] commented 9 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

kyleprice commented 9 months ago

Doesn't look like the link to the line that I think is the issue was added so here it is:

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/syslogexporter/exporter.go#L73

crobert-1 commented 9 months ago

Hello @kyleprice, thanks for filing this issue. The line you referenced is not relevant to your issue, it's simply used for the exporter to batch logs if you're using TCP.

You need to set the insecure option to true in the exporter's TLS settings. It will look like this:

  syslog:
    network: udp
    port: 1500
    endpoint: 10.0.0.2
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
    tls:
      insecure: true

insecure is set to false by default, which overrides the network: udp configuration option and always ends up connecting via tcp. This is expected behavior.

To code owners: Maybe we want to make this more clear in the README for anyone less familiar with these settings?

andrzej-stencel commented 9 months ago

Yeah I think the current behavior is very confusing. At the very least, we should document it.

I think we should change the implementation to not use the TLS configuration when UDP is used - or to put it the other way around, only use TLS configuration when TCP is being used. What do you think @crobert-1 @kkujawa-sumo @rnishtala-sumo?

rnishtala-sumo commented 9 months ago

Yes, I agree that it makes sense to remove TLS configuration for UDP since TLS typically operates over TCP.

kyleprice commented 9 months ago

Thanks, that fixed it. Appreciate the help!

kasia-kujawa commented 7 months ago

The issue was fixed in #31202, it can be closed.