Observability dev service LGTM not working : "Connection refused: getsockopt: localhost/127.0.0.1:4317"

[omasseau]

Describe the bug

Hello,

I just followed the documentation (https://quarkus.io/guides/observability-devservices-lgtm) to set up the observability dev service (based on OTel-LGTM).

The container is correctly started but it seems quarkus cannot connect to it. I get this error: 10:06:59,906 SEVERE [io.qu.op.ru.ex.ot.VertxGrpcExporter] (vert.x-eventloop-thread-4) Failed to export spans. The request could not be executed. Full error message: Connection refused: getsockopt: localhost/127.0.0.1:4317

When I look at the ports binding of the dev service container I can see indeed that there is no binding for port 4317 :

And here are the logs in quarkus when the dev service is started :

10:04:36,327 INFO  [io.qu.ob.te.Lg.docker] (build-79) Container docker.io/grafana/otel-lgtm:0.4.0 is starting: 52211bceca20ea75cf5c7f490027a6bacfbcc3b7f92c2cb8f7820be8c32f0872
10:04:36,807 INFO  [or.te.co.wa.st.HttpWaitStrategy] (build-79) /sweet_rubin: Waiting for 60 seconds for URL: http://localhost:55974/ (where port 55974 maps to container port 3000)
10:04:40,906 INFO  [io.qu.ob.te.Lg.docker] (build-79) Container docker.io/grafana/otel-lgtm:0.4.0 started in PT5.9585031S
10:04:40,907 INFO  [io.qu.ob.de.ObservabilityDevServiceProcessor] (build-79) Dev Service Lgtm started, config: {quarkus.otel-collector.url=localhost:55975, quarkus.grafana.url=localhost:55974}
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
10:04:41,469 WARN  [io.qu.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.grafana.url" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
10:04:41,469 WARN  [io.qu.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.otel-collector.url" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo

Expected behavior

Connection to LGTM container should work

Actual behavior

Connection to LGTM container is not working

How to Reproduce?

1) Add these dependencies in pom.xml :

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-observability-devservices-lgtm</artifactId>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-opentelemetry</artifactId>
        </dependency>

2) Start quarkus in dev mode

3) Make a request to a REST endpoint

4) See the error in the logs

Output of uname -a or ver

No response

Output of java -version

21.0.3

Quarkus version or git rev

3.12.3

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot[bot]]

/cc @geoand (devservices), @stuartwdouglas (devservices)

[geoand]

cc @brunobat @alesj

[alesj]

Strange ... as we have tests for this ... let me have a look

[alesj]

Ah, this is VertxGrpcExporter, which seems to send the spans at the wrong port ... is there a way to change that port to 4318?

Since that's where LGTM port is by default

    /**
     * The port on which LGTM's OTLP port will be exposed.
     */
    @WithDefault("4318")
    int otlpPort();

[alesj]

Or try

quarkus.observability.lgtm.otlp-port=4317

[brunobat]

This happens because, for some reason this is not picked up by the OpenTelemetry extension: https://github.com/quarkusio/quarkus/blob/bdadc09ef484f712b83e6ca5f71f9420130c6e5b/extensions/observability-devservices/testlibs/devresource-lgtm/src/main/java/io/quarkus/observability/devresource/lgtm/LgtmResource.java#L64

[kdubb]

@brunobat It seems to be hardcoded: https://github.com/quarkusio/quarkus/blob/47db79776008f9d9bfd6bfd4fa7ddaf4b428fa9d/extensions/observability-devservices/testcontainers/src/main/java/io/quarkus/observability/testcontainers/LgtmContainer.java#L42

[kdubb]

After setting:

quarkus.observability.lgtm.otlp-port=4317

I still see the error:

2024-09-08 21:30:45,599 WARN  [io.mic.reg.otl.OtlpMeterRegistry] (otlp-metrics-publisher-12) Failed to publish metrics. Server responded with HTTP status code -1 and body <no response body>

I think this is because the exporter is set to http/protobuf and the port it's using is the GRPC port (4317).

Given that the LGTM docs explicitly state:

The quarkus.otel.exporter.otlp.protocol is set to http/protobuf.

This appears to be on purpuse because http/protobuf should is on port 4318.

--

So.. it seems like the real problem is that the container isn't publishing the 4318 port.

[alesj]

Did you try latest upstream? This should be fixed there ...

[alesj]

... if you then leave it all to existing (hardcoded) defaults ...

[kdubb]

How do I do that? Change the image version?

[alesj]

How do I do that? Change the image version?

No, the LGTM image is all fine. You need to change / use the latest (upstream) Quarkus -- which is of course a bit trickier, since you need to build it yourself, and then use this funky 999-SNAPSHOT version ...