When this particular error happens for the solver, it needs to be killed and restarted. What happens, is the memory allocation for the nvidia driver is messed up -- it can execute kernels but just gets garbage, and then fails. On the next task, it then tries again, but fails, and so on. There is no way to get the kernels working again until the program is killed and restarted.
I will see if there are better workarounds once the problem is detected, or at least tell it to stop working entirely and have a watchdog process restart it.
When this particular error happens for the solver, it needs to be killed and restarted. What happens, is the memory allocation for the nvidia driver is messed up -- it can execute kernels but just gets garbage, and then fails. On the next task, it then tries again, but fails, and so on. There is no way to get the kernels working again until the program is killed and restarted.
I will see if there are better workarounds once the problem is detected, or at least tell it to stop working entirely and have a watchdog process restart it.