open-telemetry / opentelemetry-java-instrumentation

OpenTelemetry auto-instrumentation and instrumentation libraries for Java
https://opentelemetry.io
Apache License 2.0
1.99k stars 869 forks source link

insecure_skip_verify not working #6609

Open anilsamuel opened 2 years ago

anilsamuel commented 2 years ago

Describe the bug As per the documentation at opentelemetry-collector, insecure_skip_verify property will skip verifying the certificate. But the javaagent is producing error message below Sep 13, 2022 5:55:52 PM AEST INFO [com.boomi.container.plugin.BasePluginManager updateLoadedServer] Starting Plugin service 'Groovy 1.5.x Scripting Engine' (SCRIPT_ENGINE_GROOVY) Sep 13, 2022 5:55:53 PM AEST STDERR [com.boomi.logging.StdLogger$LoggingOutputStream flush] [otel.javaagent 2022-09-13 17:55:53:615 +1000] [OkHttp https://xxxxxxxxxxxxxxxxxxx/...] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: Hostname xxxxxxxxxxxxxx not verified: certificate: sha256/A5HyDJwg/bLJz7BpNinIAsLReIyo8OokSejz2yAx+GY= DN: CN=xxxxxxxxxx subjectAltNames: [xxxxxxxxxxxxxxxxxxx]

Steps to reproduce

What did you expect to see? Expected no error as above and to see relevant messages at the OTLP host

What did you see instead? Error messages as below Sep 13, 2022 5:55:53 PM AEST STDERR [com.boomi.logging.StdLogger$LoggingOutputStream flush] [otel.javaagent 2022-09-13 17:55:53:615 +1000] [OkHttp https://xxxxxxxxxxxxxxxxxxx/...] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: Hostname xxxxxxxxxxxxxx not verified: certificate: sha256/A5HyDJwg/bLJz7BpNinIAsLReIyo8OokSejz2yAx+GY= DN: CN=xxxxxxxxxx subjectAltNames: [xxxxxxxxxxxxxxxxxxx]

What version are you using? 1.13.1-aws from https://github.com/aws-observability/aws-otel-java-instrumentation

Environment OS: "Amazon Linux 2 Runtime (if different from JDK above): openjdk version "11.0.15" 2022-04-19 LTS OpenJDK Runtime Environment Corretto-11.0.15.9.1 (build 11.0.15+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.15.9.1 (build 11.0.15+9-LTS, mixed mode)

laurit commented 2 years ago

otel.exporter.otlp.insecure and otel.exporter.otlp.insecure_skip_verify are not supported by otel java sdk, see https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md for supported flags.

anilsamuel commented 2 years ago

otel.exporter.otlp.insecure and otel.exporter.otlp.insecure_skip_verify are not supported by otel java sdk, see https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md for supported flags.

Thanks Lauri. Hence I think our only option to use TLS for the comms is to use otel.exporter.otlp.certificate to specify the HTTPS/TLS certificate from OTLP endpoint. Right ?

trask commented 2 years ago

@anilsamuel I believe you could also import the certificate into your java keystore

anilsamuel commented 2 years ago

I have following steps to test with certificate as suggested by @trask but still not working; could you please advise.

HOST_PORT=dev-telemetry.services.xx:443
CERT_FILE=$(echo $HOST_PORT | sed 's/:/ /g' | awk '{print $1}').crt
#extract host certificates
openssl s_client -showcerts -verify 5 -connect $HOST_PORT < /dev/null 2> /dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; print}' > $CERT_FILE

java -Dotel.exporter.otlp.endpoint=https://$HOST_PORT  -Dotel.exporter.otlp.certificate=$CERT_FILE -Dotel.exporter.otlp.protocol=grpc -Dotel.exporter=otlp -Dotel.instrumentation.methods.include=Count[countChars] -Dotel.metrics.exporter=none -Dotel.resource.attributes=service.namespace=boomiruntime,service.name=boomimolecule,host.name=$(hostname),resource.type=cloud,service.group=alpha -javaagent:/opt/boomi/installers/aws-opentelemetry-agent.jar -cp . Count Count.java
trask commented 1 year ago

@anilsamuel were you able to resolve this? if not, probably we would need a standalone repro in order to help investigate it further