Running Testcontainers with Rancher Desktop and Apple Silicon fails

[therealjsie]

Actual Behavior

Running a test using Testcontainers on a MacBook Pro M1 leads to exceptions and a failed test with Rancher Desktop. Using the same setup with Docker Desktop does not result in a failed test.

Steps to Reproduce

1. Use a 2021 MacBook Pro with M1 Chip.
2. Check out the repo: https://github.com/therealjsie/rdtc-test
3. Run mvn verify with Rancher Desktop (Kubernetes disabled, using dockerd runtime)
4. Should result in a failed test.

Result

Failing logs with Rancher Desktop: repo-postgres-simple-test-macos-m1.log

Expected Behavior

Successful logs with Docker Desktop: repo-postgres-simple-test-macos-m1-docker-desktop.log

Additional Information

No response

Rancher Desktop Version

1.4.1

Rancher Desktop K8s Version

disabled

Which container runtime are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

macOs Monterey 12.4

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

N/A

Windows User Only

No response

[INeedMySpace]

I have the similar issues with flaky tests running on my M1 MacBook Pro (macOS Monterey 12.5) / Rancher Desktop 1.5.0 / testcontainers 1.17.3. Running well with Docker Desktop.

[ZombieSocrates]

Same as the prior commenter, I believe I experienced these issues with tests running on an M1 MacBook Pro (macOS Monterey 12.3)/ Rancher Desktop 1.5.0/ and two separate versions of testcontainers, 1.15.3 and 1.17.1.

As soon as I installed Docker Desktop, I could run the tests I needed to ( via Gradle, if that's important) on both of the above testcontainers.

[edeandrea]

Same here - https://github.com/quarkusio/quarkus/issues/25682

[overheadhunter]

I don't think this issue is related to the CPU architecture. I'm experiencing the same problem (well the linked Quarkus Issue) on an Intel Mac.

[adroste]

Observation: If I disable RYUK with export TESTCONTAINERS_RYUK_DISABLED="true" it works like 3 out of 4 times on my M1 MBP. Still, not a solution.

[resilva87]

Things have worked out for me when I enabled the option Allow Rancher Desktop to acquire administrative credentials (sudo access) in Preferences > Application > Behavior menu

[adroste]

@resilva87 Isn't that the default mode? (It is for me)

[resilva87]

@adroste For me at least it was not enabled. I'm running Rancher Desktop v1.6.1

[apara]

I am running Rancher Desktop v1.6.2 and still have "Connection Refused" errors when talking to TestContainers embedded Postgres. Docker Desktop seems to work fine, but errors persist with Rancher Desktop. The test mentioned here re-creates the problem well.

Here is the specific error I am seeing in my tests:

java.sql.SQLException: org.postgresql.util.PSQLException: Connection to localhost:49312 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. Caused by: org.postgresql.util.PSQLException: Connection to localhost:49312 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. Caused by: java.net.ConnectException: Connection refused

Oddly, when I manually test the connection while images are running, I am able to connect from Command Prompt.

[mobiuscog]

Maybe this helps some others, but looking at a similar problem, I found this to work with the provided example under Colima which exhibited the same problem, adding in a wait strategy to ensure the external port was mapped:

                .withPassword("PWD")
                .waitingFor(Wait.forListeningPort()))

[NagarajuKoppu]

Hi, I have similar problem with "MAC-Rancher Desktop-Test Container" combination. The intermittent exceptions of "Connection Failures" are the biggest challenge. Below changes would solve your problems.

1. Rancher-desktop would remove "localhost" from hosts configuration in MAC. So, we need to use "127.0.0.1" in the container URL's

2. Make use of "Startables" utility from test containers rather waiting for listener ports before stating the container

   private static Network network = Network.newNetwork();
   private static final PostgreSQLContainer postgreSQLContainer;
   
   static {
       postgreSQLContainer = new PostgreSQLContainer<>(DockerImageName.parse("postgres:11.1"))
               .withNetwork(network)
               .withDatabaseName("test_container_postgres")
               .withUsername("test")
               .withPassword("test");
       postgreSQLContainer.start();
   }
   
   @DynamicPropertySource
   public static void setDynamicProps(DynamicPropertyRegistry registry) {
       Startables.deepStart(postgreSQLContainer).join();
       registry.add("spring.r2dbc.url", () -> "r2dbc:postgresql://127.0.0.1:" + postgreSQLContainer.getFirstMappedPort() + "/" + postgreSQLContainer.getDatabaseName());
       registry.add("spring.r2dbc.username", postgreSQLContainer::getUsername);
       registry.add("spring.r2dbc.password", postgreSQLContainer::getPassword);
   }

[vaibsaraswat]

I am facing the same issue.

[INFO ] 2023-05-03 18:59:05 [main ] [/] [] 0:440 - Container testcontainers/sshd:1.1.0 is starting: 2e7992a0f868d11f0f947ed4a863db7a5c1b5b4be2b8503036cadc413fd33296 [ERROR] 2023-05-03 19:00:06 [main ] [/] [] 0:540 - Could not start container org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (localhost ports: [49154] should be listening)

Rancher Desktop v1.8.1 MacOS 13.3.1 M1 Test container v1.18.0

Do we have a solution or a workaround?

[jandubois]

The mvn verify command works for me on Intel, but fails on M1 because ports don't seem to be forwarded correctly. Not sure yet why. I could work around it by providing the port of the VM explicit:

export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show rd0 | awk '/inet / {sub("/.*",""); print $2}')

Output:

...
[INFO] --- jar:3.3.0:jar (default-jar) @ testcontainers-java-repro ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.565 s
[INFO] Finished at: 2023-09-13T17:07:54-07:00
[INFO] ------------------------------------------------------------------------

This requires that you run Rancher Desktop with admin access enabled, so it will allocate a routable IP:

[jandubois]

What I suspect is that the M1 machines are too fast. It takes up to 3 seconds for the port to be forwarded to the host, and if the app checks to connect to the port on localhost before the forwarding is in place, it will fail.

My Intel machine is slower, so the forward may be already created by the time the connection is attempted.

BTW, if you are using VZ emulation, then the interface name is vznat instead of rd0 (requires Rancher Desktop 1.10):

export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

[jandubois]

If you are using VZ emulation (requires macOS 13.3+ Ventura), then you can also get it to work without administrative access by exporting 2 more settings:

export DOCKER_HOST=unix://$HOME/.rd/docker.sock
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

This is possible because vzNAT networking doesn't require admin access to allocate a routable IP address. It is automatically enabled with VZ emulation, but only in the just release Rancher Desktop 1.10.0.

[pdrosos]

If you are using VZ emulation (requires macOS 13.3+ Ventura), then you can also get it to work without administrative access by exporting 2 more settings:

export DOCKER_HOST=unix://$HOME/.rd/docker.sock
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

This is possible because vzNAT networking doesn't require admin access to allocate a routable IP address. It is automatically enabled with VZ emulation, but only in the just release Rancher Desktop 1.10.0.

I have the same problems with Rancher Desktop 1.10.0 and testcontainers 1.19.0 on Macbook Pro M1 with macOS Ventura:

o.t.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:32782

and also can't connect to the PostgreSQL container (connection refused).

The same application works with Docker Desktop for Mac.

I tried everything suggested in this issue - default Rancher Desktop emulation settings and enabled administrative access, VZ emulation with the additional environment variables, unfortunately nothing works so far.

[jandubois]

o.t.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:32782

I don't think you have set the environment variables correctly if the error message still mentions localhost. It should have connected to the IP address specified in TESTCONTAINERS_HOST_OVERRIDE. Are you sure you set end exported all the variables?

[pdrosos]

o.t.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:32782

I don't think you have set the environment variables correctly if the error message still mentions localhost. It should have connected to the IP address specified in TESTCONTAINERS_HOST_OVERRIDE. Are you sure you set end exported all the variables?

I had them, it finally worked after reinstalling Rancher Desktop and restarted the computer, with VZ emulation without administrative credentials and the environment variables in .zshrc

export PATH=$PATH:$HOME/.rd/bin
export DOCKER_HOST=unix://$HOME/.rd/docker.sock
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

Thank you very much for the help!

[udf2457]

@pdrosos is this still working for you ? I'm running Rancher Desktop, with VZ, without administrative credentials, and despite using your env vars its still not working.

EDIT: Aah, I think I found the missing item. Default engine for Rancher is containerd, I changed this to docker in settings and it seems to work. Can you confirm this is your setting too @pdrosos ?

[pdrosos]

@pdrosos is this still working for you ? I'm running Rancher Desktop, with VZ, without administrative credentials, and despite using your env vars its still not working.

EDIT: Aah, I think I found the missing item. Default engine for Rancher is containerd, I changed this to docker in settings and it seems to work. Can you confirm this is your setting too @pdrosos ?

udf2457 Yes, you should use dockerd (moby) container engine, along with the environment variables above and VZ emulation.

[pdrosos]

@jandubois for some reason this line stopped working, the inner command doesn't return anything: export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

The command rdctl shell ip a show vznat shows this:

3: vznat: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:f8:ae:7f brd ff:ff:ff:ff:ff:ff
    inet6 fd20:3cc5:b420:de7b:5055:55ff:fef8:ae7f/64 scope global dynamic flags 100
       valid_lft 2591945sec preferred_lft 604745sec
    inet6 fe80::5055:55ff:fef8:ae7f/64 scope link
       valid_lft forever preferred_lft forever

Do you know what might be the reason?

[jandubois]

@pdrosos I see that your vzNAT interface only has an IPv6 address, but not an IPv4 one, which is the reason the command doesn't return anything.

I don't know right now why there would be no IPv4 address though. I thought I saw an issue about this before, but can't find anything right now.

[udf2457]

@pdrosos Its still working for me.

What version of MacOS and Rancher Desktop are you on ?

[jandubois]

I doubt it has anything to do with the macOS or Rancher Desktop versions, but is some local system issue, as this is all managed by vmnet.

I would try to reboot the machine, just to see if that fixes it, given that it has worked on the same machine before. While generally unsatisfying, it would at least maybe unblock you.

[pdrosos]

Hi @jandubois, @udf2457

Unfortunately restart didn't help, it was the first I tried. I am with Rancher Desktop 1.10.0 (latest), MacBook Pro M1, MacOS Sonoma. It's strange indeed why this suddenly stopped working and there's no IPv4 any more, I have no idea why.

[jandubois]

I've found 2 people on Slack that have experienced something similar, and it is still unknown why this is happening. One person claimed it happend with 13.4.0, was fixed in 13.4.1, but happened again with 13.5.0, but the other person seemed to have experienced it with 13.4.1 as well.

Now @pdrosos reports it happening in 15.?? as well, so it seems like an ongoing problem.

I don't know if this can be worked around with the upcoming user-v2 networking support in Lima, but we won't be able to support this until 1.12 late in the year, as the work on 1.11 has to wrap up pretty soon.

[udf2457]

@pdrosos

Ah, Sonoma. I'm still on Ventura (13.6).

So we really need some more feedback from Sonoma users I guess, just to rule that possibility out.

[Fonzerelli13]

Has the same issue with Rancher Desktop and Test Containers. My config is: MacBook Pro with M1 chip, MacOS Sonoma. Rancher Desktop 1.10.0 Administrative access allowed, QEMU Virtual machine type.

With such settings I am getting the same error "Connection refused". But interesting thing here is that if I stop in debug in place where it tries to connect to the database, wait for several seconds and then proceed the test then it works fine. With VZ emulation have the same issues as @pdrosos regarding IPV6 adddress.

[pdrosos]

Hi @Fonzerelli13

I had to add additional Wait.forListeningPort() strategy to my PostgreSQL testcontainer in order to work on my MacBook Pro M1. This was related to another Lima specific problem with a delay in port forwarding: https://github.com/abiosoft/colima/issues/71#issuecomment-979516106

private static final PostgreSQLContainer<?> POSTGRESQL_CONTAINER = new PostgreSQLContainer<>(DockerImageName.parse("postgres:15.4"))
      .withUsername(USERNAME)
      .withPassword(PASSWORD)
      .withDatabaseName(SCHEMA)
      .waitingFor(new WaitAllStrategy()
          .withStrategy(Wait.forListeningPort())
          .withStrategy(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2)
              .withStartupTimeout(Duration.ofSeconds(60))
          )
      );

[Fonzerelli13]

@pdrosos thank you for your advice. At the moment looks that Test Containers works correctly after adding permanent enviroment variables into .zprofile file and rebooting my mac.

[stefanscheidt]

@jandubois Any updates regarding rdctl shell ip a show rd0 only returning inet6 addresses?

On my M1 mac with macOS 14.1.1 (23B81) and Rancher Desktop 1.11.0 I only see this:

❯ rdctl shell ip a show rd0
4: rd0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:96:30:ad brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5055:55ff:fe96:30ad/64 scope link
       valid_lft forever preferred_lft forever

[stefanscheidt]

@jandubois It seems to depend on the network the host machine is in. This morning I was sitting in the office connected to the WLAN there and had the no-inet-issue mentioned just above. Now I'm at home in my own WLAN and see this:

❯ rdctl shell ip a show rd0
4: rd0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:96:30:ad brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.109/24 brd 192.168.178.255 scope global rd0
       valid_lft forever preferred_lft forever
    inet6 fe80::5055:55ff:fe96:30ad/64 scope link
       valid_lft forever preferred_lft forever

Thus export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show rd0 | awk '/inet / {sub("/.*",""); print $2}') works again ...

[bsideup]

I understand that this comment might be a little bit unethical to make given that this is an issue in Rancher Desktop's repo, but I wanted to point out that (at least as a temporary solution), Testcontainers Desktop now comes with an embedded runtime that has a built-in solution for this (detecting the IP while the VZ backend is used).

P.S. we (Testcontainers) love Rancher Desktop and even mention it specifically as one of the available local runtime options in Testcontainers Desktop! ❤️ I was just following this thread and did not feel good about all these Testcontainers users who are having issues 🙌

[stefanscheidt]

@jandubois I can reproduce my problem: When being at home, the Rancher Desktop VM can obtain an IPv4 address, when connected with the company WIFI, it can't - maybe some certificate issues, I don't know - I'm more or less a network noob.

Is there a way to not use a "bridged setup" (which I assume RD does) but a "NAT setup" for the VM and still have a reachable IP address for Testcontainers?

[stefanscheidt]

And another update: I tried to use VZ emulation without administrative rights in my home office network to avoid the bridged network. But then no inet shows up, only inet6:

$ rdctl shell ip a show vznat
3: vznat: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:1a:a7:50 brd ff:ff:ff:ff:ff:ff
    inet6 fddf:917c:6f8d:8527:5055:55ff:fe1a:a750/64 scope global dynamic flags 100
       valid_lft 2591994sec preferred_lft 604794sec
    inet6 fe80::5055:55ff:fe1a:a750/64 scope link
       valid_lft forever preferred_lft forever

Thus Testcontainers will not work, because we can not set TESTCONTAINERS_HOST_OVERRIDE as shown here

export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

[sairaghavak]

I tried this approach https://github.com/rancher-sandbox/rancher-desktop/issues/2609#issuecomment-1718556068

It worked without giving sudo access to rancher.

Again, the issue popped up and the fix was not permanent.

Later, I tried the wait strategy as suggested here: https://github.com/rancher-sandbox/rancher-desktop/issues/2609#issuecomment-1788871956

It worked in my case with and without corporate VPN and verified in build pipeline as well, it works! Thanks.

MacOs: Sonoma 14.3.1 Rancher Desktop Version: 1.12.3

[kari-jumo]

Running rdctl shell ip a show vznat I get ip: can't find device 'vznat'

Rancher Desktop 1.13.1 for aarch64 on M3

[slayton]

@kari-jumo I ran into the same thing until I disabled run as administrator and ensured Kubernetes was disabled.

[mwisnicki]

Still does not work.

Rancher 1.15.0 in default configuration with Kubernetes disabled.

Following https://docs.rancherdesktop.io/how-to-guides/using-testcontainers/

git clone https://github.com/testcontainers/testcontainers-java-repro
cd testcontainers-java-repro
./mvnw verify

First failure:

 Tests in error: 
 demonstration(org.testcontainers.repro.ReproExampleTest): Could not find a valid Docker environment. Please see logs and 
 check configuration

Website forgot to mention DOCKER_HOST but ok

export DOCKER_HOST=unix://$HOME/.rd/docker.sock

Still fails:

. org.testcontainers.containers.ContainerLaunchException: Container startup failed for image testcontainers/ryuk:0.6.0
 Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
. Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
. Caused by: com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such container: 60a9216f94ea7800b824fd5419aac9f1d8c7c0765ab1248f2d0d82afdb65d72a"}

export TESTCONTAINERS_RYUK_DISABLED=true

BUILD SUCCESS

[mwisnicki]

Got it to work with VZ instead of QEMU, same as https://github.com/rancher-sandbox/rancher-desktop/issues/2609#issuecomment-1738432949

Also no IPv4 on vznat while on home wifi (but with Tailwind vpn)