The deprecated runner version fails after job assignment not after start to listen for jobs

[enescakir]

Describe the bug In the previous versions, the runner used to fail prior to the job assignment and kept the job as queued if the runner version is deprecated. However, in the latest version, the failure occurs after the job has been picked, resulting in an immediate job failure. Waiting in queue for a runner with the new version was a better option.

√ Connected to GitHub
Current runner version: '2.316.1'
2024-06-27 15:51:23Z: Listening for Jobs
2024-06-27 15:51:26Z: Running job: test
2024-06-27 15:51:32Z: Job test completed with result: Canceled
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...

To Reproduce Assign a job to runner with version 2.316.1

Expected behavior Exit the runner due to deprecation before picking a job

Example log from older versions

√ Connected to GitHub
Current runner version: '2.310.2'
2024-02-03 15:06:22Z: Listening for Jobs
An error occured: Runner version v2.310.2 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...

[MuhammadElsaeed]

I am facing the same issue, how come the image is deprecated? it was released just last month. Are we supposed to update them every now and then?

[enescakir]

I am facing the same issue, how come the image is deprecated? it was released just last month. Are we supposed to update them every now and then?

There are lot of discussions on this topic. I used to version deprecation within a month

https://github.com/actions/runner/issues/3332 https://github.com/actions/runner/discussions/3235

But the runner was failing before picking a job, so the job was waiting in the queued state for 24 hours. We were able to provision a new runner with the new version. But with the latest version, the runner started to fail after picking a job, so it causes job failure.

[luketomlinson]

Hi @enescakir thanks for raising this. Do you have a link to a failed run you could share?

[enescakir]

Hi @enescakir thanks for raising this. Do you have a link to a failed run you could share?

@luketomlinson It's the run which I previously shared logs.

https://github.com/ubicloud/github-e2e-test-workflows/actions/runs/9699711981/job/26769354466

[enescakir]

@luketomlinson The interesting part is that not all jobs are failing. We use different GitHub Apps for local development, and they manage to run jobs with this older version. However, the production GitHub app can't run with this version. I don't know its details but the GitHub backend might have a certain check for the assigned job, and it may have determined that this job can't run with this runner version.

[luketomlinson]

Thanks @enescakir, that's all helpful info, I'm looking into it.

[dt-napier]

Got same problem today. We have 3 runner scale sets in one namespace, with same docker image, but they target different repositories. And only one fails with this error:

[RUNNER 2024-06-28 15:03:27Z INFO MessageListener] VssConnection created
[RUNNER 2024-06-28 15:03:27Z INFO Terminal] WRITE LINE: 

√ Connected to GitHub
[RUNNER 2024-06-28 15:03:27Z INFO Terminal] WRITE LINE: 

[RUNNER 2024-06-28 15:03:27Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:27Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:28Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-06-28 15:03:28Z INFO MessageListener] Session created.
[RUNNER 2024-06-28 15:03:28Z INFO Terminal] WRITE LINE: Current runner version: '2.316.1'
Current runner version: '2.316.1'
[RUNNER 2024-06-28 15:03:28Z INFO Terminal] WRITE LINE: 2024-06-28 15:03:28Z: Listening for Jobs
2024-06-28 15:03:28Z: Listening for Jobs
[RUNNER 2024-06-28 15:03:28Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-06-28 15:03:28Z INFO MessageListener] BrokerMigration message received. Polling Broker for messages...
[RUNNER 2024-06-28 15:03:28Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:28Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-06-28 15:03:28Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=138c0456-6361-41aa-9a21-28bdecdc5eaa&status=Online&runnerVersion=2.316.1&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-06-28 15:03:28Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-06-28 15:03:28Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-06-28 15:03:29Z ERR  Terminal] WRITE ERROR: An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
[RUNNER 2024-06-28 15:03:29Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
[RUNNER 2024-06-28 15:03:29Z ERR  Listener] --- End of stack trace from previous location ---
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxRetryAttemptsCount, Func`2 shouldRetry)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Program.MainAsync(IHostContext context, String[] args)
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...```

[luketomlinson]

This issue should be fixed now. Old runners should no longer be able to be assigned jobs. @enescakir @MuhammadElsaeed @dt-napier Can you confirm you're no longer seeing this?

[enescakir]

Thanks to your prompt solution, @luketomlinson, it's back to functioning as it used to. Was this problem related to the backend services?

√ Connected to GitHub
Current runner version: '2.316.1'
2024-07-02 07:01:03Z: Listening for Jobs
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.Exiting runner...

[luketomlinson]

Hey @enescakir, yep it was a backend issue. I'm going to close this out for now, but please open an issue if you see it again!

[Jungzhang]

Hi, I am facing the same issue @luketomlinson

[Jungzhang]

Upgrade to 2.317.0 is ok.