The original code stops the clock as soon as torch finishes issuing GPU operations, but the results wouldn't actually be ready at that point, so it's an underestimate. We need an explicit sync to correctly time end-to-end execution of the model.
Testing on flux-dev, 1024x768 resolution, 28-steps, on H100 SXM.
Before: 5520ms
After: 6040ms
Exact delta depends on how much the CPU is running ahead of the GPU, which varies significantly across systems. This also means that timings should be more stable after this change.
Torch CUDA operations are async by default.
The original code stops the clock as soon as torch finishes issuing GPU operations, but the results wouldn't actually be ready at that point, so it's an underestimate. We need an explicit sync to correctly time end-to-end execution of the model.
Testing on flux-dev, 1024x768 resolution, 28-steps, on H100 SXM. Before: 5520ms After: 6040ms
Exact delta depends on how much the CPU is running ahead of the GPU, which varies significantly across systems. This also means that timings should be more stable after this change.
credit to @donglinz for finding this bug!