Over the course of training, the average step speed can increase by up to 50% (from 0.4 seconds/step to 0.6/seconds per step, on my rig), before falling again. The speed decrease is weirdly consistent, where it can increase to 0.6 for 1000 steps (for example), before falling to 0.4 again, for another 1000 steps. It's not a one-off issue; it's a trend that ebbs and flows.
I suspect the issue might lie in the dataloaders, but don't know for sure. For example, if training batches are smaller, the system may need to spend more CPU cycles processing chunks of data - which could lead to such a slowdown. There might be a better way to pre-compute batches, or to process them in a background thread, but I haven't had time to troubleshoot this yet.
Over the course of training, the average step speed can increase by up to 50% (from 0.4 seconds/step to 0.6/seconds per step, on my rig), before falling again. The speed decrease is weirdly consistent, where it can increase to 0.6 for 1000 steps (for example), before falling to 0.4 again, for another 1000 steps. It's not a one-off issue; it's a trend that ebbs and flows.
I suspect the issue might lie in the dataloaders, but don't know for sure. For example, if training batches are smaller, the system may need to spend more CPU cycles processing chunks of data - which could lead to such a slowdown. There might be a better way to pre-compute batches, or to process them in a background thread, but I haven't had time to troubleshoot this yet.