jeroen-mostert
commented
1 month ago Some ideas.
There are a bunch of environment variables you can set to enable debugging, the most prominent of which is AMD_LOG_LEVEL. On the highest level this spews out a lot of logging if not masked with AMD_LOG_MASK, and a lot of it is super technical, so it's not generally that helpful. But it might track the problem down to the last thing that was being attempted. As that page describes, you can also use rocgdb to get a backtrace on a crash, but while powerful, gdb is a bit of an ogre if you're not already familiar with it.
I'm not familiar with how the setup on mobile works, but try to ensure that the GPU you're doing compute on is not also the one driving the primary display. This should work fine (generally the worst that can happen is running out of memory, not a crash), but it's still noise you're better off without. If some fancy desktop compositor is tickling the card in a way that's not appropriately combined with compute, this could cause the driver to choke. That would still generally be a driver bug as that should never trigger a reset, but such things do happen.
Lastly, although this doesn't really sound like a hardware problem given the random nature of the crash, you can still try installing corectrl (or building it from this SDK if your distro doesn't offer it) and see if limiting the GPU by power or clock speed improves things, since compute tends to hammer things much harder than 3D apps. Note that not all GPU models actually support this kind of tuning (the iGPU in desktop processors does not, for example) and you may need a kernel parameter to unlock the support in the first place. (Note that the wiki talks about overclocking, but it's really about unlocking voltage/speed parameters that also allow underclocking, which can help give you better performance by not driving the thing up to the thermal limit constantly. I do not recommend manually writing things to /sys as described in the wiki, just use a tool.)
jrl290
lamikr
I'm still having this random GPU Hang on my 7840U (gfx1103) and not on my 6800U (forced to gfx1030):
HW Exception by GPU node-1 (Agent handle: 0x5ab48bbcc960) reason :GPU HangI've been racking my head to figure out what's causing it. Deleting sections of my code. Trying to build a minimum crashing sample to provide. But sometimes it takes running many iterations of the processing I'm doing and sometimes it crashes right up front. There's a lot of code to go through, so I'm still trying narrow things down. But my guess is that the crash occurs as a result of the state of the GPU rather than the actual instruction, which makes things much trickier.
Maybe there's something much more obvious to you or an easier way to track down the issue
Some commands it has crashed on:
torch.stft(x, n_fft=self.n_fft, hop_length=self.hop_length, window=window, center=True,return_complex=False).to(device)torch.zeros([*batch_dims, c, n - f, t]).to(device)torch.istft(x, n_fft=self.n_fft, hop_length=self.hop_length, window=window, center=True)torch.cuda.synchronize()Here's the kernel log with a few of these crashes