Open awni opened 1 month ago
Hmm this has nothing to do with distributed processing unfortunately. The following also fails with the same error
for i in {0..3}; do python <path to script> & done; wait
I guess since this is somewhat rare.. and there isn't much if anything we can do in MLX, I will close this for now :.
Some possible work-arounds:
The internal error signifies that the GPU kernel timed out. One way to fix that is to decrease the size of the operations:
For example if you are using LoRA, try
Another possible fix is to decrease the number of operations per command buffer, this may work in some cases but may also slow things down. You can do that like so:
MLX_MAX_OPS_PER_BUFFER=1 <training command here>
Run the following with:
Fails on an M2 Ultra with: