OpenTelemetryCollector should use secure connection (TLS)

[jvanz]

During a debug session with @jordojordo we could make the communication between Kubewarden components and OpenTelemetryCollector working only after disabling TLS.

Which seems to be a bad thing to do. So, I think we need to double check if we can make it work with TLS enable and document what it is necessary for that.

[flavio]

Our collector is running as a sidecar, that means the connection between the controller and the sidecar happens on localhost. It's fine to not use HTTPs in this case.

Are you sure the issue wasn't between the collector and the target destination (like jaeger)?

[jvanz]

Our collector is running as a sidecar, that means the connection between the controller and the sidecar happens on localhost. It's fine to not use HTTPs in this case.

Okay

Are you sure the issue wasn't between the collector and the target destination (like jaeger)?

I don't think so. Because after disabling the TLS we start to see the traces in Jaeger.

[flavio]

Ok, then let's ensure we do not force the TLS connection to be used between the policy-server and the collector that runs as a sidecar :+1:

[jvanz]

AFAICS, our code configure a connection without TLS: https://github.com/kubewarden/kubewarden-controller/blob/main/internal/pkg/metrics/metrics.go#L28-L30

Maybe we need to change the default configuration.

[flavio]

In the default deployment, the controller connects to a collector that runs as a sidecar, hence on localhost. It's fine in this use case to force a HTTP connection instead of a HTTPS one. The sidecar won't have certificates, it will listen on HTTP.

[flavio]

Closing right now, might revisit in the future

[jvanz]

For future readers, you can workaround this issue configuring the collector to allow insecure connections: https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md