Open mrc0mmand opened 2 years ago
Hey @mrc0mmand,
Thank you for reporting it and providing follow-up information. I have triaged this issue and added it to the board. :relaxed:
@nikola-jokic, for what it's worth, my team is experiencing this issue with our self-hosted runners.
I provided a link and screenshot to our most recently problematic job run, which has continued to show as "pending" despite the job successfully completing 20 hours prior.
This issue is intermittent but we are also facing the same, the step (subtask) remains in pending state forever even when the job completed steps which are downstream. Are we tracking the issue ? @nikola-jokic
Any updates ? @nikola-jokic
Is this issue being tracked by anyone ? @nikola-jokic
I am experiencing same issue - it keeps running even after the completion - Any update on this?
We are facing the same issue fairly consistently.
It's been a long time. We are facing the same issue! Any update?
Describe the bug I one of our systemd repositories, I came across a strange issue where if a step fails, the runner correctly reports the step failed, but then it remains stuck in the "running" state for several hours until it's killed by the global watchdog. Also, killing the job manually in this state is also... strange, since hitting "Cancel workflow" once doesn't work, one has to hit it many times to make it kill the job.
This feels like it's the same issue as https://github.com/actions/runner/issues/700 (and given the last comment from @rubencodes it might still be unresolved).
To Reproduce Steps to reproduce the behavior: I still can't pinpoint what exactly causes this issue, but I can reproduce it reliably in this PR: https://github.com/redhat-plumbers/systemd-rhel8/pull/302, where the problematic steps looks like
sudo -E script.sh -> docker exec ... ninja -C build test
.Expected behavior The job should report a failure immediately when a step fails (or in a reasonable time frame).
Runner Version and Platform
2.294.0
OS of the machine running the runner? OSX/Windows/Linux/... Ubuntu 22.04
What's not working?
The job reports it failed, e.g.:
but then remains stuck for several hours in "running" state until it's eventually cancelled by the global watchdog:
The job itself: https://github.com/redhat-plumbers/systemd-rhel8/actions/runs/2690828471/attempts/2
A bit of further debugging shows that the information about the failed process correctly "bubbles up" the tree all the way to the runner, which then gets stuck: