I have done some profiling and the proposal part of the model (Generalized_RCNN.RPN) takes nearly 20% of the time during GPU inference for me.
I looked into it and found that there is still some numpy code in there, I think if this code is replaced with PyTorch GPU code, the inference can be significantly sped up on newer GPUs.
I have done some more tests and it appears that most of the time is spent on transfer since these layers take significantly less time during CPU inference than during GPU inference.
I have done some profiling and the proposal part of the model (
Generalized_RCNN.RPN
) takes nearly 20% of the time during GPU inference for me.I looked into it and found that there is still some numpy code in there, I think if this code is replaced with PyTorch GPU code, the inference can be significantly sped up on newer GPUs.