ARM runner has been stuck for multiple days

msys2 / msys2-autobuild

🏭🏭🏭🏭🏭🏭🏭🏭

https://packages.msys2.org/queue

MIT License

36 stars 25 forks source link

ARM runner has been stuck for multiple days #80

Open Biswa96 opened 11 months ago

Biswa96 commented 11 months ago

This CI job is running for days https://github.com/msys2-arm/msys2-autobuild/actions/runs/6508662089

@jeremyd2019

jeremyd2019 commented 11 months ago

It managed to hang up right as I was leaving on a long weekend trip, and I didn't notice until I got back. I wanted to get a fresh runner going anyway, for the latest Windows Updates, but was waiting until I got back to try to avoid any issues while I was gone :grin:. New runner is going now

Biswa96 commented 11 months ago

@jeremyd2019 Would you like to check if this CI job is stuck again https://github.com/msys2-arm/msys2-autobuild/actions/runs/6570301888 ?

lazka commented 11 months ago

@jeremyd2019 https://github.com/msys2-arm/msys2-autobuild/actions/runs/6714121489/job/18246869719

lazka commented 10 months ago

@jeremyd2019 likely stuck https://github.com/msys2-arm/msys2-autobuild/actions/runs/6980352526/job/18995436393

jeremyd2019 commented 10 months ago

I've been ruminating on the idea of some sort of 'watchdog' to detect and kill stuck pacman processes automatically, but I haven't settled on the best language/technology to do so. It seems like python would be most convenient since autobuild is already python, I could put a background thread like I did to try polling the token, but I'm not familiar with process querying/killing modules.

What I've got so far is a cygwin commands to get the cygwin pid of the process I want to kill (what I really want is the child pacman process, this gets the newest pacman process older than 1800 seconds)

pgrep -xn -O 1800 pacman

coupled with the script I already had (because when stuck in this state cygwin kill is not sufficient) https://github.com/jeremyd2019/winautoconfig/blob/master/msys2-runner-setup/setupscripts/wkill.sh

Biswa96 commented 10 months ago

It would be a bit clear if the reason of such CI failure is explained.

jeremyd2019 commented 10 months ago

lost power, so any lack of runner in the near future will be due to that

power is back

lazka commented 9 months ago

@jeremyd2019 https://github.com/msys2-arm/msys2-autobuild/actions/runs/7355161710

lazka commented 7 months ago

@jeremyd2019 https://github.com/msys2-arm/msys2-autobuild/actions/runs/7862352714

jeremyd2019 commented 6 months ago

unstuck it. the powershell variant in https://github.com/git-for-windows/git-for-windows-automation/pull/61#issuecomment-2014464189 was intriguing, it seems like it could be close to being turned into a 'watchdog', would just need to also query CreationDate field to see any pacman processes that have been running a long time (like a half hour? or hour?), and then arrange for it to run continuously (scheduled task?). Of course, I'd much rather get whatever bug is causing this fixed...

lazka commented 5 months ago

https://github.com/msys2-arm/msys2-autobuild/actions/runs/8938789079/job/24553638109

jeremyd2019 commented 4 months ago

There's a stuck job now, but it doesn't seem to be the runner this time. Probably something on Github's end.

Biswa96 commented 3 months ago

stuck again https://github.com/msys2-arm/msys2-autobuild/actions/runs/9739288128

jeremyd2019 commented 3 months ago

This seems to be a different issue. I think maybe the machine rebooted. I did a quick check and didn't notice any excess packages installed.

Biswa96 commented 2 months ago

stuck again https://github.com/msys2-arm/msys2-autobuild/actions/runs/10108768753

lazka commented 1 month ago

"echo: write error: No space left on device"