search

actions / runner-images

GitHub Actions runner images
MIT License
9.4k stars 2.91k forks source link

ubuntu-latest `20240609.1.0` is forcing IPv6 on Verdaccio #10061

Open cortinico opened 3 weeks ago

cortinico commented 3 weeks ago

Description

Hey all, @cortinico from React Native here.

I've noticed that the latest deployment of the ubuntu-latest runner on GitHub actions (version 20240609.1.0) is breaking our CI on React Native causing it to take 6+ hours to complete.

This happend as the runner was updated from 20240603.1.0 to 20240609.1.0 (see here https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240609.1).

✅ Successful Run: https://github.com/facebook/react-native/actions/runs/9473329852/job/26101842381 ❌ Faulty Run (6h+): https://github.com/facebook/react-native/actions/runs/9484523671/job/26136577410

The problem seems to be related to IPv4 vs IPv6. We do use Verdaccio in our CI setup to test NPM packages. For some reasons, Verdaccio is always listening on a IPv6 port since 20240609.1.0.

I'm unsure what could be causing this, but I tried all possible ways to force Verdaccio to listen on a IPv4 address/port. It seems like to always fallback to an IPv6.

Happy to have more insights on what caused this as nothing changed other than the Ubuntu Runner version.

I also created a reproducer here: https://github.com/cortinico/repro-ipv6-gha-issue/

You can see in the latest run how I'm running Verdaccio and using net-stat to show the open ports: https://github.com/cortinico/repro-ipv6-gha-issue/actions/runs/9503903039/job/26195373761 Seems like Verdaccio is listening on a IPv6 port: Screenshot 2024-06-13 at 18.16.35.png

Platforms affected

Runner images affected

Image version and build link

20240609.1.0

Is it regression?

20240603.1.0

Expected behavior

Processes should honor the port/address configuration provided

Actual behavior

Process is ignoring the configuration and always opening an IPv6 port/address.

Repro steps

Reproducer is here: https://github.com/cortinico/repro-ipv6-gha-issue/

ArminArdekani commented 3 weeks ago

Surprised this hasn't been flagged as a incident yet by Github Actions, considering ipv6 isn't supported. It's affecting our CI jobs today too.

The culprit is when they updated the runners to use Docker v26 which has the loopback: https://docs.docker.com/engine/release-notes/26.0/#bug-fixes-and-enhancements-2

cortinico commented 3 weeks ago

@lakshminarayana02 @vidyasagarnimmagaddi any news on what's the ETA for this fix? Also are there any workaround?

cortinico commented 3 weeks ago

For context this is broken among ubuntu-20.04, ubuntu-latest and ubuntu-24.04

vidyasagarnimmagaddi commented 3 weeks ago

Hi @cortinico , we are analyzing the issue, will revert ASAP

vidyasagarnimmagaddi commented 3 weeks ago

Hi @cortinico , We tried reproducing the same.In file verdaccio.yml , add listen:

kindly revert if any issues.

cortinico commented 3 weeks ago

We tried reproducing the same.In file verdaccio.yml , add listen:

Hi @vidyasagarnimmagaddi Thanks for getting back to me. I'm not sure I fully understand your answer.

As you see from my reproducer, when calling netstat -ano the open port still shows up as IPv6.

Moreover it seems also other users had similar issues (@ansgarm and @ArminArdekani)

erik-bershel commented 3 weeks ago

Hey @cortinico!

Please use a workaround config for the tool with using of IPv4 address directly. For example:

listen:
   0.0.0.0:4873

The same time we'll investigate why the default resolver for localhost was changed to IPv6 instead of IPv4.

ansgarm commented 3 weeks ago

Hi everyone 👋

I can confirm that explicitly configuring an IPv4 address worked for us 👍 Thanks for the workaround instructions, @erik-bershel!

cortinico commented 3 weeks ago

Thanks for the clarification @erik-bershel 👍

erik-bershel commented 3 weeks ago

Hey @cortinico

After further investigation, I can say the following. Correct me if you see any discrepancies.

In general, the latest Docker changes have actually caught up with similar changes that have existed for a long time, including on our Ubuntu runners. If you try to run the ping localhost command you will find that it accesses the address ::1. In general, the problem is not even related to the fact that Docker added the localhost alias to the ::1 address, but to the fact that by default Ubuntu and many other systems consider IPv6 addresses to have higher priority when resolving names.

You have several ways to work around this problem in Docker:

Unfortunately, Docker updates constantly bring new calls, yeah. But right now we would not want to roll back these changes for a number of reasons:

cc @ansgarm @ArminArdekani @samjarrett

ArminArdekani commented 3 weeks ago

@erik-bershel Our problem from this was that we could not communicate with the docker containers HEALTHCHECK endpoint that was specified in the Dockerfile with localhost. So we managed to fix it by replacing localhost with 0.0.0.0 address in the dockerfile.

cortinico commented 3 weeks ago

You have several ways to work around this problem in Docker:

Thanks for the workaround @erik-bershel 👍 this unblocks us

rogertinsley commented 3 weeks ago

Had the same issue with maildev with the healthcheck in the Dockerfile:

https://github.com/maildev/maildev/blob/357a20edcd205413d3590aedb8fcd7c97563c40d/Dockerfile#L23

Ended up adding an environment variable to unblock me:

  maildev:
    hostname: maildev
    image: maildev/maildev
    ports:
      - 1080:1080
      - 1025:1025
    environment:
      MAILDEV_IP: '::'