Down Detector - Githubissues

gpu-mode / discord-cluster-manager

Hook up GPUs to your Discord channel and start running jobs via DMs!

4 stars 2 forks source link

Down Detector #34

Open msaroufim opened 1 day ago

msaroufim commented 1 day ago

Nov 21: AMD jobs stuck in queue https://github.com/gpu-mode/discord-cluster-manager/actions/runs/11958587509
As of Nov 20: NVIDIA jobs stuck in queue forever https://github.com/gpu-mode/discord-cluster-manager/actions/runs/11944808187
Nov 20: AMD jobs can't find a hip GPU https://github.com/gpu-mode/discord-cluster-manager/actions/runs/11945340557 - cc @saienduri - fixed

saienduri commented 1 day ago

Logged into the runner and simple rocm commands weren't working. Possible that one of the jobs crashed rocm. Anyways, gave it a reboot and should be good now