Using insecure: true does not skip CA validation in otlphttp exporter

[sgamelin]

Component(s)

No response

Describe the issue you're reporting

When configuring the opentelemetry collector with an otlphttp exporter with a configuration as follows:

  otlphttp:
    endpoint: https://self-signed.badssl.com
    sending_queue:
      enabled: false
    tls:
      insecure: true

The following log is generated by the collector after producing a trace:

{"level":"info","ts":1687894452.5027428,"msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"otlphttp","error":"failed to make an HTTP request: Post \"https://self-signed.badssl.com/v1/traces\": tls: failed to verify certificate: x509: certificate signed by unknown authority","interval":"41.977312225s"}

It seems that also using http protocol in the endpoint results in the same issue:

  otlphttp:
    endpoint: http://self-signed.badssl.com
    sending_queue:
      enabled: false
    tls:
      insecure: true

{"level":"info","ts":1687897096.7850964,"msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"otlphttp","error":"failed to make an HTTP request: Post \"https://self-signed.badssl.com/v1/traces/v1/traces\": tls: failed to verify certificate: x509: certificate signed by unknown authority","interval":"6.263633711s"}

According to the documentation in opentelemetry-collector, the following is stated:

    // In gRPC when set to true, this is used to disable the client transport security.
    // See https://godoc.org/google.golang.org/grpc#WithInsecure.
    // In HTTP, this disables verifying the server's certificate chain and host name
    // (InsecureSkipVerify in the tls Config). Please refer to
    // https://godoc.org/crypto/tls#Config for more information.
    // (optional, default false)

This appears to imply that having insecure: true should disable TLS (including certificate verification). However, this is not the behaviour observed with the otlphttp exporter.

Can it please be confirmed if this is expected? Thanks.

Additional environment details:

• Collector version: 0.79.0

[sgamelin]

I realize now that the otlphttp exporter is part of https://github.com/open-telemetry/opentelemetry-collector, if I need to create the issue there, please let me know.

[andrzej-stencel]

I think the documentation for the TLS settings is not quite correct. In my testing, for the OTLP/HTTP exporter, it's the URL scheme that defines whether TLS will be used or not. If you specify http://example.com as your URL, TLS will not be used. If you specify https://example.com, TLS will be used. To connect with a self-signed certificate, use insecure_skip_verify: true instead of insecure: true.

exporters:
  otlphttp:
    endpoint: https://self-signed.badssl.com
    sending_queue:
      enabled: false
    tls:
      insecure_skip_verify: true

Let me know if this works for you.

[sgamelin]

Hi @astencel-sumo, thanks for your response, and yes, that is also the configuration that I am using for now, which works as expected. From my above example:

It seems that also using http protocol in the endpoint results in the same issue:
...
  otlphttp:
    endpoint: http://self-signed.badssl.com
    sending_queue:
      enabled: false
    tls:
      insecure: true

The scheme in this URL is HTTP, however, TLS verification is being enforced. From the logs it seems that the request is upgraded to HTTPS along the way:

... "error":"failed to make an HTTP request: Post \"https://self-signed.badssl.com/v1/traces/v1/traces\": tls: failed to verify certificate: x509 ...

So in brief, if the endpoint's URL scheme is HTTPS or the request ends up being upgraded to HTTPS along the way, the following setting to disable TLS:

tls:
  insecure: true

will have no effect, and the following needs to be used to skip CA verification:

tls:
  insecure_skip_verify: true

I am wondering if the documentation of this exporter (and perhaps other exporters that may behave in a similar way) should be updated to reflect this.

[atoulme]

Please feel free to offer a documentation update.

[chenlujjj]

Misled by the doc, wasting nearly one hour on it ..