search

actions / runner

The Runner for GitHub Actions :rocket:
https://github.com/features/actions
MIT License
4.91k stars 965 forks source link

Forbidden Runner version v2.317.0 is deprecated and cannot receive messages #3381

Closed jonapich closed 4 months ago

jonapich commented 4 months ago

This happened to us on 2.317.0, which is the latest version at time of writing:

An error occured: Error: Forbidden Runner version v2.317.0 is deprecated and cannot receive messages.

It happened twice. Timestamps in UTC.

image image

past occurence: https://github.com/actions/runner/issues/3366#issuecomment-2204514441

luketomlinson commented 4 months ago

hi @jonapich Do you have your runner's session id? Also, it seems like those logs you posted are from yesterday?

jonapich commented 4 months ago

@luketomlinson it did happen yesterday; sorry for the mix up. Good catch 😅

I think 8baf0469-317a-49fa-bc3e-7b835570c4f5 would be the session id.

jonapich commented 4 months ago

Full log line: 

GET request to https://broker.actions.githubusercontent.com/message?sessionId=8baf0469-317a-49fa-bc3e-7b835570c4f5&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
luketomlinson commented 4 months ago

Thanks @jonapich. This was rolled back yesterday. I'm going to close this one out.

dtakis commented 4 months ago

@luketomlinson can we have more details about this? We faced the same issue with v2.316.1 and upgraded to v2.317.0 to resolve this problem. What was rolled back?

luketomlinson commented 4 months ago

👋 @dtakis

Currently, if you register your runner with the disableupdate flag, if your runner is on an a version that is not the latest, we force you to upgrade to the latest if that version is > 30 days old. ChangeLog

The bug that was rolled back was incorrectly performing that age check on the latest version.

If you register a runner without the disableupdate flag, you should never run into this.

jacobwgillespie commented 3 months ago

@luketomlinson We are currently seeing this error again, here's an example session:

[2024-07-26 17:05:02Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=cfa9c468-c485-4c87-9c52-ae63a51668a6&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden

Since v2.318.0 was just released one hour ago, perhaps this has broken the logic?

luketomlinson commented 3 months ago

Taking a look @jacobwgillespie

jake-setford commented 3 months ago

@luketomlinson We are currently seeing this error again, here's an example session:

[2024-07-26 17:05:02Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=cfa9c468-c485-4c87-9c52-ae63a51668a6&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden

Since v2.318.0 was just released one hour ago, perhaps this has broken the logic?

Getting this aswell on all our runners using 2.317.0.

andyspiers commented 3 months ago

Also getting this on many repos, but not all.

We have alarms firing and production deployments that are not happening.

We have to run with updates disabled (because we run ephemeral runners in Kubernetes which start, run one job then exit) but we check for newer runner software every day and rebuild our images if a new base image is found. However given that the only newer version is 2 hours old, there is no newer container base image for us to build from.

Surely blocking the N-1 release when N is only 2 hours old is an error on Github's part?

luketomlinson commented 3 months ago

👋 Apologies for this error on our part. It should be rolled back now. Restarting runners should get it back up and rolling again.

jake-setford commented 3 months ago

@luketomlinson can confirm all runners up again thanks for sorting.

jacobwgillespie commented 3 months ago

Can confirm ours are working as well, thank you for the fast turnaround ❤️

malwinders-evertz commented 2 months ago

@luketomlinson This is happening again with the latest v2.319.1. Did try restaring the runner few times but it stops with-in seconds and throws this message. Sep 03 10:07:48 streampro-ghrunner runsvc.sh[7881]: An error occured: Error: Forbidden Runner version v2.319.1 is deprecated and cannot receive messages.

luketomlinson commented 2 months ago

Hi @malwinders-evertz, do you have any runner logs, specifically the /message url that the runner is polling against?

malwinders-evertz commented 2 months ago

Yes, [2024-09-03 22:24:35Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=5f0250c2-e019-433e-ac52-d61720ceacef&status=Online&runnerVersion=2.319.1&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: Forbidden


[2024-09-03 22:24:35Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Error: Forbidden Runner version v2.319.1 is deprecated and cannot receive messages.
luketomlinson commented 2 months ago

@malwinders-evertz Do you happen anything in your infrastructure (proxies/firewalls or what not) that might be throwing a 403 in between the runner and GitHub? I don't see that request make it to our servers, but I'm still digging. It's possible a firewall might be rejecting broker.actions.githubusercontent.com with a 403, which is incorrectly getting interpreted as an old runner version by the Runner itself.

malwinders-evertz commented 2 months ago

@luketomlinson yes, we do have a VPN setup in our infrastructure, This exact same issue occured earlier as reported, around August 13. Later on August 14 there was a github outage for almost all the services and when that outage got resolved our runner started working as expected.

luketomlinson commented 2 months ago

Thanks @malwinders-evertz, can you confirm that broker.actions.githubusercontent.com is allowed within your VPN? I'm not seeing this as widespread right now, and I'm suspicious that I don't see the request make it to us.

malwinders-evertz commented 2 months ago

Yes, i will verify that, first thing in the morning as i have to reach out to our IT team regarding this and right now no one from IT is availabe. Will get back to you in the morning.

malwinders-evertz commented 2 months ago

@luketomlinson I confirmed with IT, the above address was not allowed over our VPN, once they allowed it and we restarted the runner service things seems working as expected. Thanks for the quick Response. Still wondering why it was throwing Error: Forbidden Runner version v2.319.1 is deprecated and cannot receive messages

luketomlinson commented 2 months ago

Thanks @malwinders-evertz. I think that's an error reporting bug on the runner side. It's interpreting any 403 as a deprecated version error, which is not correct. I'm looking into fixing that.

omgitsbillryan commented 2 months ago

Count me in as a datapoint. Running 2.317.0 and started seeing this error at 9:00am this morning.

pavanpatilingenovis commented 2 months ago

Had the same issue here.

Sep 18 15:34:29 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: Started running service Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: √ Connected to GitHub Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: Current runner version: '2.317.0' Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: 2024-09-18 15:34:31Z: Listening for Jobs Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.

I updated '2.319.0' ...... still same issue

devAl1 commented 2 months ago

I'm using Helm's latest version 0.9.3 and get this issue. Here are some logs, sigh.

√ Connected to GitHub
[RUNNER 2024-09-19 16:52:42Z INFO Terminal] WRITE LINE:

[RUNNER 2024-09-19 16:52:42Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-09-19 16:52:42Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-09-19 16:52:43Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-09-19 16:52:43Z INFO MessageListener] Session created.
[RUNNER 2024-09-19 16:52:43Z INFO Terminal] WRITE LINE: Current runner version: '2.319.0'
Current runner version: '2.319.0'
[RUNNER 2024-09-19 16:52:43Z INFO Terminal] WRITE LINE: 2024-09-19 16:52:43Z: Listening for Jobs
2024-09-19 16:52:43Z: Listening for Jobs
[RUNNER 2024-09-19 16:52:43Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-09-19 16:52:43Z ERR  GitHubActionsService] GET request to https://pipelinesghubeus10.actions.githubusercontent.com/ziWsyY3jNadAyEohLfZPgIwCZR6CEEED8DQ8cYkMpF1XbwcOFv/_apis/distributedtask/pools/1/messages?sessionId=418d28f7-7aaf-43a9-862c-699130c5d94e&status=Online&runnerVersion=2.319.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-09-19 16:52:43Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-09-19 16:52:43Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-09-19 16:52:43Z ERR  Terminal] WRITE ERROR: An error occured: Runner version v2.319.0 is deprecated and cannot receive messages.
luketomlinson commented 2 months ago

👋 From our docs, for a given version, once a newer version is made available, there are 30 days until that version is deprecated.

v2.319.1 is the latest version. Anything older than that is now outside the supported window.

makashu commented 1 month ago

Snippet from the Support Policy notes the following:

We support (at maximum) 2 GA images and 1 beta image at a time. We begin the deprecation process of the oldest image label once the newest OS image label has been released to GA

So the documentation here https://docs.github.com/en/enterprise-cloud@latest/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#controlling-runner-software-updates-on-self-hosted-runners seems conflicting this doc https://github.com/actions/runner-images/?tab=readme-ov-file#software-and-image-support?

Does these docs need to be in-sync?

devAl1 commented 1 month ago

Got it. Turns out we were using a custom runner based upon the runner image. So we just regenerated the runner image and we were good. I initially missed the minor version being off.

V-know commented 1 month ago

We encountered the same issue sine 2 days ago, the error log is:

ERR GitHubActionsService] GET request to https://pipelinesghubeus22.actions.githubusercontent.com/pLDjbMqlMDBG5S74r6lIYrddOvCnPtmrY70Us9frMAg06mRSm0/_apis/distributedtask/pools/15/messages?sessionId=4a970bbe-5fa5-458e-905e-c284c29dd85e&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-09-25 10:43:26Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-09-25 10:43:26Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-09-25 10:43:26Z ERR Terminal] WRITE ERROR: An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-25 10:43:26Z ERR Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)

Just now we upgaded the runner version from v2.317.0 to v2.319.1, all errors disappeared. Our runners started working as expected.