Closed jaycedowell closed 11 months ago
I haven't seen this on any node other than the head node.
This could be a GPU problem of some sort. If I run the T-engines as packet capture only then I don't get the lockup. The lockup message also always (?) references a python3
process which points to the T-engine's GPU usage. Should we try to clean the pins/reseat the GPUs/NICs on the next site trip?
As I read into this it sounds like a driver/firmware issue. Updating the BIOS didn't fix cetus for this same issue. However, most of the forum threads on this topic talk about changing nvidia drivers to try to fix it.
Yeah, worth a shot. Both cetus and ndp servers are on 525.125.06.
There's an interesting pattern emerging here. I don't get soft lockups on the T-engine if I don't use the PFB inverter (even running all four beams). I do get the soft lockups on ndp1 and ndp2 if I try large buffers in the cuda_host
memory space. This points to something related to large memory allocations.
I'm now not seeing this problem even with the PFB running. I'm going to say that this is just a symptom of the non-optimal memory use that we had.
These seem to have started on ndp now that the system is under some load. I saw them on cetus as well but never came up with a good solution.