The deprecated runner version fails after job assignment not after start to listen for jobs

enescakir commented 3 days ago

Describe the bug In the previous versions, the runner used to fail prior to the job assignment and kept the job as queued if the runner version is deprecated. However, in the latest version, the failure occurs after the job has been picked, resulting in an immediate job failure. Waiting in queue for a runner with the new version was a better option.

√ Connected to GitHub
Current runner version: '2.316.1'
2024-06-27 15:51:23Z: Listening for Jobs
2024-06-27 15:51:26Z: Running job: test
2024-06-27 15:51:32Z: Job test completed with result: Canceled
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...

To Reproduce Assign a job to runner with version 2.316.1

Expected behavior Exit the runner due to deprecation before picking a job

Example log from older versions

√ Connected to GitHub
Current runner version: '2.310.2'
2024-02-03 15:06:22Z: Listening for Jobs
An error occured: Runner version v2.310.2 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...

MuhammadElsaeed commented 3 days ago

I am facing the same issue, how come the image is deprecated? it was released just last month. Are we supposed to update them every now and then?

enescakir commented 3 days ago

I am facing the same issue, how come the image is deprecated? it was released just last month. Are we supposed to update them every now and then?

There are lot of discussions on this topic. I used to version deprecation within a month

https://github.com/actions/runner/issues/3332 https://github.com/actions/runner/discussions/3235

But the runner was failing before picking a job, so the job was waiting in the queued state for 24 hours. We were able to provision a new runner with the new version. But with the latest version, the runner started to fail after picking a job, so it causes job failure.

luketomlinson commented 3 days ago

Hi @enescakir thanks for raising this. Do you have a link to a failed run you could share?

enescakir commented 3 days ago

Hi @enescakir thanks for raising this. Do you have a link to a failed run you could share?

@luketomlinson It's the run which I previously shared logs.

https://github.com/ubicloud/github-e2e-test-workflows/actions/runs/9699711981/job/26769354466

enescakir commented 3 days ago

@luketomlinson The interesting part is that not all jobs are failing. We use different GitHub Apps for local development, and they manage to run jobs with this older version. However, the production GitHub app can't run with this version. I don't know its details but the GitHub backend might have a certain check for the assigned job, and it may have determined that this job can't run with this runner version.

luketomlinson commented 3 days ago

Thanks @enescakir, that's all helpful info, I'm looking into it.

dt-napier commented 3 days ago

Got same problem today. We have 3 runner scale sets in one namespace, with same docker image, but they target different repositories. And only one fails with this error:


[RUNNER 2024-06-28 15:03:27Z INFO MessageListener] VssConnection created
[RUNNER 2024-06-28 15:03:27Z INFO Terminal] WRITE LINE: 

√ Connected to GitHub
[RUNNER 2024-06-28 15:03:27Z INFO Terminal] WRITE LINE: 

[RUNNER 2024-06-28 15:03:27Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:27Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:28Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-06-28 15:03:28Z INFO MessageListener] Session created.
[RUNNER 2024-06-28 15:03:28Z INFO Terminal] WRITE LINE: Current runner version: '2.316.1'
Current runner version: '2.316.1'
[RUNNER 2024-06-28 15:03:28Z INFO Terminal] WRITE LINE: 2024-06-28 15:03:28Z: Listening for Jobs
2024-06-28 15:03:28Z: Listening for Jobs
[RUNNER 2024-06-28 15:03:28Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-06-28 15:03:28Z INFO MessageListener] BrokerMigration message received. Polling Broker for messages...
[RUNNER 2024-06-28 15:03:28Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:28Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-06-28 15:03:28Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=138c0456-6361-41aa-9a21-28bdecdc5eaa&status=Online&runnerVersion=2.316.1&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-06-28 15:03:28Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-06-28 15:03:28Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-06-28 15:03:29Z ERR  Terminal] WRITE ERROR: An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
[RUNNER 2024-06-28 15:03:29Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
[RUNNER 2024-06-28 15:03:29Z ERR  Listener] --- End of stack trace from previous location ---
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxRetryAttemptsCount, Func`2 shouldRetry)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Program.MainAsync(IHostContext context, String[] args)
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...```

actions / runner

The deprecated runner version fails after job assignment not after start to listen for jobs #3366