search

actions / runner

The Runner for GitHub Actions :rocket:
https://github.com/features/actions
MIT License
4.75k stars 923 forks source link

Actions Runner fails on IPv6 only host #3138

Open alsutton opened 6 months ago

alsutton commented 6 months ago

Describe the bug On an IPv6 only host a self-hosted action runner fails to register itself as online due to a failed network request to a URL with an IPv4 only address.

Context: Many hosting companies are now charging for IPv4 addresses (most recently AWS), so having IPv6-only self-hosted runners is desirable from an operational cost perspective.

To Reproduce

  1. Configure a self-hosted action-runner on an IPv4/v6 host.
  2. Reconfigure the host to be IPv6 only.
  3. Try to start the action-runner.
  4. Runner fails to register and reports error in the logs (below)

Expected behavior I expected the runner to start and register as online.

Runner Version and Platform

Version of your runner? 2.312.0

OS of the machine running the runner? OSX/Windows/Linux/... Linux (Debian 12)

What's not working?

Start-up and registration flow requires IPv4 to connect to a URL (pipelinesghubeus4.actions.githubusercontent.com)

nslookup for host indicates no IPv6 address;

$ nslookup pipelinesghubeus4.actions.githubusercontent.com
Server:  UnKnown
Address:  fdc9:993e:6df:0:3e9e:c7ff:fe92:2ee8

Non-authoritative answer:
Name:    pipelinesghubeus4.actions.githubusercontent.com
Address:  20.232.252.48

Job Log Output

Runner_20240208-075819-utc.log, potential token redacted;

[2024-02-08 07:59:02Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2024-02-08 07:59:02Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 11.01 seconds.
[2024-02-08 07:59:11Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 12.97 seconds.
[2024-02-08 07:59:12Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 12.91 seconds.
[2024-02-08 07:59:13Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 12.829 seconds.
[2024-02-08 07:59:24Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 17.574 seconds.
[2024-02-08 07:59:25Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 18.057 seconds.
[2024-02-08 07:59:25Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 18.337 seconds.
[2024-02-08 07:59:41Z ERR  GitHubActionsService] Attempt 4 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The maximum number of attempts has been reached.
[2024-02-08 07:59:41Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2024-02-08 07:59:41Z INFO RunnerServer] Catch exception during connect. 3 attempt left.
[2024-02-08 07:59:41Z ERR  RunnerServer] System.Net.Http.HttpRequestException: Network is unreachable (pipelinesghubeus4.actions.githubusercontent.com:443)
 ---> System.Net.Sockets.SocketException (101): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32
lastChangeId, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at GitHub.Runner.Common.RunnerService.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
[2024-02-08 07:59:41Z ERR  RunnerServer] #####################################################
[2024-02-08 07:59:41Z ERR  RunnerServer] System.Net.Sockets.SocketException (101): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
[2024-02-08 07:59:41Z INFO GitHubActionsService] Starting operation Location.GetConnectionData

Runner and Worker's Diagnostic Logs

Logs given above

alsutton commented 6 months ago

Possible duplicate of #2840. Is there any news on progress?

DracoBlue commented 6 months ago

I am interested in this, too.

undergroundwires commented 5 months ago

Here's my workaround (open-source and documented) that I hope that can help you too:

After days of research and trial/error, this is how I got this working:

  1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
  2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
  3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: https://github.com/undergroundwires/privacy.sexy/commit/52fadcd6177ed06216be9c67dad57192ae02a4f9

KeithYeh commented 4 months ago

We faced the same issue when migrating our self-hosted runners to IPv6-only VPC on AWS. BTW, their CloudFormationInit endpoint does not support IPv6 either.

esantora commented 1 month ago

The runner needs a configurable switch to use IPv4 only in dual-stack systems until GitHub actually supports IPv6.