Keycloak DevService not considered healthy when running in docker shared network mode

flo-02-mu commented 2 years ago

Describe the bug

When running the QuarkusIntegrationTest against the keycloak dev service using docker network (e.g. by setting quarkus.container-image.build=true) the http based health check fails:

2021-12-05 19:07:47,923 ERROR [🐳 .io/.0.2]] (build-7) Could not start container: java.lang.IllegalArgumentException: Requested port (8080) is not mapped
    at org.testcontainers.containers.ContainerState.getMappedPort(ContainerState.java:153)
    at java.base/java.util.Optional.map(Optional.java:265)
    at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:177)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
    at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.lambda$startContainer$0(KeycloakDevServicesProcessor.java:303)

I tried a quick fix by adding:

@Override
public Integer getMappedPort(int originalPort) {
    if (useSharedNetwork) {                      
        return KEYCLOAK_PORT;
    } else {
        return super.getMappedPort(originalPort);
    }
}

near https://github.com/quarkusio/quarkus/blob/main/extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java#L489

This solves the above exception, but it reveals another issue: The health check still fails because it tries to reach the /auth url on the internal network address (e.g. http://keycloak-8yVNc:8080/auth):

Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://keycloak-8yVNc:8080/auth should return HTTP 200)
    at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:264)

This seems logical, since the healthcheck is made from outside the docker network. I also checked how other devservices are doing this, but none is using the HttpWaitStrategy.

Expected behavior

Keycloak DevService gets recognized as healthy and the test can continue

Actual behavior

Exception, see above

How to Reproduce?

https://github.com/flo-02-mu/security-openid-connect-quickstart-networkerror

mvn verify

Output of `uname -a` or `ver`

20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64

Output of `java -version`

openjdk 12.0.2 2019-07-16 OpenJDK Runtime Environment (build 12.0.2+10) OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.5.1.Final (also tried on latest master)

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 2019-08-27T17:06:16+02:00)

Additional information

No response

quarkus-bot[bot] commented 2 years ago

/cc @jmartisk, @pedroigor, @sberyozkin, @xstefank

geoand commented 2 years ago

@manstis any idea?

manstis commented 2 years ago

I'll take a look.

geoand commented 2 years ago

Thanks a lot!

manstis commented 2 years ago

I suspect the use-case is much like I had with Kafka.. when I needed it available on both the shared network and docker host network. I'll look into when I'm at my desk a little later.

manstis commented 2 years ago

@geoand OK, so a couple of things:-

I can confirm the reported behaviour.
It is not a regression following my recent change :champagne:
It could be an issue with keycloak only being exposed on the internal Docker network.....
.... but if I add quarkus.container-image.builder=docker to the application.properties file it works OK.
So.... there's something else going on (that I know nothing about).. the reproducer uses jlib as the (default?) image builder.

Unfortunately @geoand I cannot be of much more help as there needs to be an understanding of how the different image builders work and how they are meant to interact with TestContainers (which is IIUC only Docker based)...

geoand commented 2 years ago

Thanks a lot for the analysis @manstis.

flo-02-mu commented 2 years ago

Isn't the HttpWaitStrategy the issue? That one is used in the keycloak-dev-service. For all other waitstrategies, the container is somehow directly queried (e.g. for log messages). But for the HttpWaitStrategy an actual http endpoint is queried. When using non-docker-network, the query succeeds, because it queries (e.g.) http://localhost:60048/auth which is a reachable URL. But when using the docker-network, it translates to http://keycloak-8yVNc:8080/auth . This URL cannot be queried from the host machine. In the past there was a log-strategy implemented for keycloak, but I guess this does not play well together with the different keycloak / keycloak-x options.

flo-02-mu commented 2 years ago

.... but if I add quarkus.container-image.builder=docker to the application.properties file it works OK.

I checked this and I can confirm it: When launching with image build = docker, the port gets mapped additionally:

CONTAINER ID   IMAGE                              COMMAND                  CREATED          STATUS          PORTS                               NAMES
533f81988f93   quay.io/keycloak/keycloak:15.0.2   "/opt/jboss/tools/do…"   13 seconds ago   Up 11 seconds   8443/tcp, 0.0.0.0:51411->8080/tcp   loving_proskuriakova

When running it with jib, no port mapping of port 8080 takes place. I guess the port mapping gets added here https://github.com/quarkusio/quarkus/blob/main/extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java#L393 But that would mean that inc ase of image.builder=docker the shared network is not used.

manstis commented 2 years ago

The problem is that when quarkus.container-image.build=true then useSharedNetwork is true and this runs instead. IMO the problem is definitely something to do with exposing KeyCloak correctly to the different containers and builders... however IDK anything about the quarkus.container-image.xxx properties (I'm just a contributor that provided a patch to something that was in the area of the useSharedNetwork flag).. We're both going to have to wait to hear what the core Quarkus development team think.

geoand commented 2 years ago

I'll try and take a look tomorrow or Wednesday

sberyozkin commented 2 years ago

Thanks @manstis @geoand, if I can help somehow let me know please, so far all the tests have been running (in the quickstarts, security-openid-connect-quickstart, security-openid-connect-web-authentication-quickstart, but also integration-tests/oidc-token-propagation ) without the shared network.

geoand commented 2 years ago

I'll take care of this one and in the meantime add an integration test that will use Keycloak in a @QuarkusIntegrationTest

quarkusio / quarkus