Closed mmuckley closed 3 years ago
I identified the cause of this issue. It was due to overhead of repeated calls to torch.set_num_threads
. Apparently these are more expensive in PyTorch 1.8.
I previously added these lines due to an observation that torchkbnufft
wouldn't respect an OMP_NUM_THREADS
environment variable. For example, if you had 8 threads on your system and you set OMP_NUM_THREADS
, torchknufft
would use all 8 threads if you didn't use the torch.set_num_threads
commands during process forks. After removing the lines, the performance of the adjoint on CPU is much better. I don't see the oversubscription issue for the adjoint, but it remains for the forward operation, so we may need to do further work on adjoint threading.
I think I'm going to release version 1.2.0 of torchkbnufft
for now to handle PyTorch 1.8 without regressions, and we can try to do more threading work for the adjoint in the future.
I am noticing the performance degradation with PyTorch 1.8 for the CPU on my home system (Windows 10, i5 8400, GTX 1660,
torchkbnufft
version 1.1.0). It looks like the GPU is relatively unaffected. The details are below. I'm not sure why this is happening yet, but I will try to look into it. If anyone has any information, feel free to post on this issue.PyTorch 1.8:
PyTorch 1.7.1: