Why C++ version libtorch so slow

[CaichaoGitHub]

my code is sample

torch::jit::getProfilingMode() = false; torch::jit::setGraphExecutorOptimize(false); torch::Tensor tensor = torch::zeros({1,3, 1280, 1280},at::kFloat); torch::jit::script::Module model = torch::jit::load("./psenet.pt"); model.eval(); std::vector<torch::jit::IValue> inputs; inputs.push_back(tensor); long long t1 = getSystemTime(); for(int i = 0; i < 8 ;++i) { auto output = model.forward(inputs).toTensor(); }

`long long t2 = getSystemTime();
printf("cost %f \n",(t2 - t1) / 8.0 / 1000.0);`

but the code is 2 times slower than pytorch version ? anyone know the problem?

cc @suo

[suo]

Hi, thank you for the report! Are you able to share the model that is seeing this issue? It's difficult to investigate without a repro. Thanks!

[CaichaoGitHub]

Hi, thank you for the report! Are you able to share the model that is seeing this issue? It's difficult to investigate without a repro. Thanks!

thank you for the reply! my model was the PSENet , which i cloned from https://github.com/whai362/PSENet , i converted it using the torch.jit.trace , and i tested it on my mac-book . the libtorch was downloaded from the official site without any modification

[suo]

One thing that I noticed is that you're running without profiling or optimizations. What happens if you turn it on, warm up the model (3-5 runs should be sufficient), then measure the avg time after those runs?

[WANGCHAO1996]

One thing that I noticed is that you're running without profiling or optimizations. What happens if you turn it on, warm up the model (3-5 runs should be sufficient), then measure the avg time after those runs?

Hello, my cuda10.2 libtorch1.7.1 model yolov5 was slow in the first two tests and the second one was the slowest and then normal. What is the reason? ![Uploading 图片.png…]()

[gemfield]

libtorch performance issue analysis: https://zhuanlan.zhihu.com/p/363319763