Open saimouli opened 2 years ago
I believe this is just an unfortunate limitation of GPU kernel initialization. There is not much you can do other than maybe running dummy inputs for however many slow inferences there are before going into your real data.
It takes a long time to do inference for the first few images. Progressively the inference time reduces. Does engine generation optimize according to the device? Any cache/changes needed to make the inference time constant for any number of iterations?