Not able to connect to Apicurio dev service when running integration tests

[edeandrea]

Describe the bug

I have an application which listens for events on Kafka and processes them. This application also uses an Apicurio schema registry. This application has integration tests which use a QuarkusTestResourceLifecycleManager to create a KafkaProducer to "drop" events onto the Kafka topic for the application to consume in the tests. The Kafka broker & Apicurio schema registry are started/provided by Dev Services.

https://github.com/quarkusio/quarkus-super-heroes/blob/main/event-statistics/src/test/java/io/quarkus/sample/superheroes/statistics/KafkaProducerResource.java is the class, and https://github.com/quarkusio/quarkus-super-heroes/tree/main/event-statistics is the application. The test resource class is reading the mp.messaging.connector.smallrye-kafka.apicurio.registry.url property provided by the Apicurio dev services and setting it's value as the SerdeConfig.REGISTRY_URL in the KafkaProducer config.

When running the integration tests against a containerized JVM version of the application (i.e. when running ./mvnw clean verify -Dquarkus.container-image.build=true) the KafkaProducer doesn't appear to be able to communicate with the Apicurio schema registry.

Expected behavior

I would expect that my integration test should be able to interact with the Apicurio schema registry started by dev services.

Actual behavior

When I try to run the IT tests I can see in the logs that it is starting the container and specifying the network:

Executing "docker run --name quarkus-integration-test-hcIks --rm -p 64917:64917 -p 8444:8444 --net=66250d06c4a45a0142a4600e9df09ed472fd8692f0be2d7a319495bb8ff486d9 --env QUARKUS_HTTP_PORT=64917 --env QUARKUS_HTTP_SSL_PORT=8444 --env TEST_URL=http://0.0.0.0:0 --env KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://kafka-mYSzY:29092,OUTSIDE://localhost:49158 --env MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL=http://kafka-Wm7Ac:8080/apis/registry/v2 quay.io/quarkus-super-heroes/event-statistics:java11-latest"

My question is - how do I use that containerNetworkId in the construction of the KafkaProducer, specifically in the SerdeConfig.REGISTRY_URL pointing to the apicurio registry? It doesn't appear that the apicurio instance is exposed outside of the container network. The tests fail with:

[ERROR] io.quarkus.sample.superheroes.statistics.endpoint.WebSocketsIT.testScenarios  Time elapsed: 10.983 s  <<< ERROR!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.net.UnknownHostException: Failed to resolve 'kafka-Wm7Ac' after 3 queries 
        at event.statistics@1.0/io.quarkus.sample.superheroes.statistics.endpoint.WebSocketsIT.testScenarios(WebSocketsIT.java:81)
Caused by: java.util.concurrent.ExecutionException: java.net.UnknownHostException: Failed to resolve 'kafka-Wm7Ac' after 3 queries 
        at event.statistics@1.0/io.quarkus.sample.superheroes.statistics.endpoint.WebSocketsIT.testScenarios(WebSocketsIT.java:81)
Caused by: java.net.UnknownHostException: Failed to resolve 'kafka-Wm7Ac' after 3 queries 

How to Reproduce?

No response

Output of uname -a or ver

Darwin edeandre-mac 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64

Output of java -version

OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.7.2.Final (but has been nagging me for a few releases now)

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /Users/edeandre/.m2/wrapper/dists/apache-maven-3.8.4-bin/52ccbt68d252mdldqsfsn03jlf/apache-maven-3.8.4
Java version: 11.0.2, vendor: Oracle Corporation, runtime: /Users/edeandre/.sdkman/candidates/java/11.0.2-open
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

Additional information

No response

[quarkus-bot[bot]]

/cc @alesj, @ozangunalp

[edeandrea]

I think the underlying problem is that the Apicurio registry is not exposed outside of the docker network? The test class (which is not running in a container) can not access the container on the docker network.

[edeandrea]

Hi just wanted to get any information about this? Even after switching my tests to use the KafkaCompanion, the integration tests are still not able to work.

Clone https://github.com/quarkusio/quarkus-super-heroes, go into either the rest-fights or event-statistics directory and run ./mvnw clean verify -Dquarkus.container-image.build=true. The integration tests fail because the KafkaCompanion producer/consumer in the ITs can't talk to apicurio.

[edeandrea]

Hi - any thoughts on this?

[cescoffier]

Are you using podman? The name of the service uses uppercase characters.

[cescoffier]

@geoand did the fix got pushed?

[geoand]

@edeandrea seems to be using a fork of superheroes that you and I work on :)

You mean the networking fix? Is so, yes, that is in main (see https://github.com/quarkusio/quarkus/pull/26422)

[edeandrea]

@cescoffier no I'm using Docker. This fails when running as a GitHub action as well.

[edeandrea]

The problem is that the test class executing the integration test uses a KafkaCompanion and is unable to reach the apicurio instance started by dev services because. My guess is that

a) the apicurio instance is not exposed outside of the container network

b) the hostname configured by dev services is specific to the container network and not reachable by the host process

I could be wrong though. I just know it doesn't work

[ozangunalp]

First of all, I am running into another issue running integration tests on a container image build. Using minikube as docker machine, the docker host is not on localhost, but on 192.168.205.4. But the test uri's injected into tests with @TestHTTPResource have localhost. I had to pass -Dtest.url='http://192.168.205.4:${quarkus.http.port}' to the command line for it to work.

I think the described error is related. For Kafka dev service it is not an issue : Kafka dev service bootstrap servers config has 2 URLs, one for internal docker network, one for external. So even though docker network hostname kafka-Wm7Ac is not resolvable from the host, the second URL is. For Apicurio, the registry URL we are getting from the application config is for internal docker network, therefore not accessible from tests running on the host.

I think this was an issue for DB dev services as well. In integration tests dev service DBs are not accessible.

[edeandrea]

Hi folks. Any progress on this?

[cescoffier]

I think @ozangunalp had a look but that was a few months ago.

[cescoffier]

Do we know if this is still an issue?

[edeandrea]

It was the last time I tried it.

[ozangunalp]

As I mentioned before this is a more global issue with with how dev services are run on integration tests. I haven't spend the time to look at it yet.

[edeandrea]

I just tried it again and can verify it is still an issue. I think though that the issue is more generic - trying to run @QuarkusIntegrationTests where the test itself needs to communicate with a dev service.

In this specific case the test uses the kafka companion to communicate with the kafka broker. The kafka broker is exposed both on the container network and outside so both the app running as a container and the integration test itself can communicate with it.

The apicurio registry though is not exposed outside the container network, so the integration test is not able to communicate with it.

08:09:43 INFO  [io.qu.te.co.DefaultDockerContainerLauncher] (main) Executing "docker run --name quarkus-integration-test-ypCCR -i --rm \
  -p 65263:65263 \
  -p 8444:8444 \
  --net=2e06fc91e5bf75ce5161c005fb1feb5c27107c9661ddc8218d513ab8e863bb28 \
  --env QUARKUS_LOG_CATEGORY__IO_QUARKUS__LEVEL=INFO \
  --env QUARKUS_HTTP_PORT=65263 \
  --env QUARKUS_HTTP_SSL_PORT=8444 \
  --env TEST_URL=http://0.0.0.0:0 \
  --env MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL=http://kafka-nbjgm:8080/apis/registry/v2 \
  --env QUARKUS_HTTP_HOST=0.0.0.0 \
  --env KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://kafka-kfz5i:29092,OUTSIDE://localhost:55011 \
  --env QUARKUS_CONTAINER_IMAGE_BUILD=true \
  --env MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_SCHEMA_REGISTRY_URL=http://kafka-nbjgm:8080/apis/ccompat/v6 \
  quay.io/quarkus-super-heroes/event-statistics:1.0"