❓ [Question] Linking error with PTQ function

jonahclarsen commented 2 years ago

❓ Question

I am getting a linking error when using torch_tensorrt::ptq::make_int8_calibrator. I am using the Windows build based on CMake, so I'm not sure if it's a problem with the way it was built, but I suspect not since I can use functions from ::torchscript just fine.

I am trying to create a barebones program to test ptq based on examples/int8/ptq/main.cpp, and I get this linker error whenever torch_tensorrt::ptq::make_int8_calibrator is used. Any help would be greatly appreciated.

Environment

PyTorch Version (e.g., 1.0): 1.11+cu113
OS (e.g., Linux): Windows 10
How you installed PyTorch (conda, pip, libtorch, source): libtorch from pytorch.org
CUDA version: 11.3

Additional context

This is the linker error that I get:

Severity Code Description Project File Line Suppression State Error LNK2019 unresolved external symbol "declspec(dllimport) class torch_tensorrt::ptq::Int8Calibrator<class nvinfer1::IInt8EntropyCalibrator2,class std::unique_ptr<class torch::data::StatelessDataLoader<class torch::data::datasets::MapDataset<class torch::data::datasets::MapDataset<class datasets::CIFAR10,struct torch::data::transforms::Normalize >,struct torch::data::transforms::Stack<struct torch::data::Example<class at::Tensor,class at::Tensor> > >,class torch::data::samplers::RandomSampler>,struct std::default_delete<class torch::data::StatelessDataLoader<class torch::data::datasets::MapDataset<class torch::data::datasets::MapDataset<class datasets::CIFAR10,struct torch::data::transforms::Normalize >,struct torch::data::transforms::Stack<struct torch::data::Example<class at::Tensor,class at::Tensor> > >,class torch::data::samplers::RandomSampler> > > > __cdecl torch_tensorrt::ptq::make_int8_calibrator<class nvinfer1::IInt8EntropyCalibrator2,class std::unique_ptr<class torch::data::StatelessDataLoader<class torch::data::datasets::MapDataset<class torch::data::datasets::MapDataset<class datasets::CIFAR10,struct torch::data::transforms::Normalize >,struct torch::data::transforms::Stack<struct torch::data::Example<class at::Tensor,class at::Tensor> > >,class torch::data::samplers::RandomSampler>,struct std::default_delete<class torch::data::StatelessDataLoader<class torch::data::datasets::MapDataset<class torch::data::datasets::MapDataset<class datasets::CIFAR10,struct torch::data::transforms::Normalize >,struct torch::data::transforms::Stack<struct torch::data::Example<class at::Tensor,class at::Tensor> > >,class torch::data::samplers::RandomSampler> > > >(class std::unique_ptr<class torch::data::StatelessDataLoader<class torch::data::datasets::MapDataset<class torch::data::datasets::MapDataset<class datasets::CIFAR10,struct torch::data::transforms::Normalize >,struct torch::data::transforms::Stack<struct torch::data::Example<class at::Tensor,class at::Tensor> > >,class torch::data::samplers::RandomSampler>,struct std::default_delete<class torch::data::StatelessDataLoader<class torch::data::datasets::MapDataset<class torch::data::datasets::MapDataset<class datasets::CIFAR10,struct torch::data::transforms::Normalize >,struct torch::data::transforms::Stack<struct torch::data::Example<class at::Tensor,class at::Tensor> > >,class torch::data::samplers::RandomSampler> > >,class std::basic_string<char,struct std::char_traits,class std::allocator > const &,bool)" (imp_??$make_int8_calibrator@VIInt8EntropyCalibrator2@nvinfer1@@V?$unique_ptr@V?$StatelessDataLoader@V?$MapDataset@V?$MapDataset@VCIFAR10@datasets@@U?$Normalize@VTensor@at@@@transforms@data@torch@@@datasets@data@torch@@U?$Stack@U?$Example@VTensor@at@@V12@@data@torch@@@transforms@34@@datasets@data@torch@@VRandomSampler@samplers@34@@data@torch@@U?$default_delete@V?$StatelessDataLoader@V?$MapDataset@V?$MapDataset@VCIFAR10@datasets@@U?$Normalize@VTensor@at@@@transforms@data@torch@@@datasets@data@torch@@U?$Stack@U?$Example@VTensor@at@@V12@@data@torch@@@transforms@34@@datasets@data@torch@@VRandomSampler@samplers@34@@data@torch@@@std@@@std@@@ptq@torch_tensorrt@@YA?AV?$Int8Calibrator@VIInt8EntropyCalibrator2@nvinfer1@@V?$unique_ptr@V?$StatelessDataLoader@V?$MapDataset@V?$MapDataset@VCIFAR10@datasets@@U?$Normalize@VTensor@at@@@transforms@data@torch@@@datasets@data@torch@@U?$Stack@U?$Example@VTensor@at@@V12@@data@torch@@@transforms@34@@datasets@data@torch@@VRandomSampler@samplers@34@@data@torch@@U?$default_delete@V?$StatelessDataLoader@V?$MapDataset@V?$MapDataset@VCIFAR10@datasets@@U?$Normalize@VTensor@at@@@transforms@data@torch@@@datasets@data@torch@@U?$Stack@U?$Example@VTensor@at@@V12@@data@torch@@@transforms@34@@datasets@data@torch@@VRandomSampler@samplers@34@@data@torch@@@std@@@std@@@01@V?$unique_ptr@V?$StatelessDataLoader@V?$MapDataset@V?$MapDataset@VCIFAR10@datasets@@U?$Normalize@VTensor@at@@@transforms@data@torch@@@datasets@data@torch@@U?$Stack@U?$Example@VTensor@at@@V12@@data@torch@@@transforms@34@@datasets@data@torch@@VRandomSampler@samplers@34@@data@torch@@U?$default_delete@V?$StatelessDataLoader@V?$MapDataset@V?$MapDataset@VCIFAR10@datasets@@U?$Normalize@VTensor@at@@@transforms@data@torch@@@datasets@data@torch@@U?$Stack@U?$Example@VTensor@at@@V12@@data@torch@@@transforms@34@@datasets@data@torch@@VRandomSampler@samplers@34@@data@torch@@@std@@@std@@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@4@_N@Z) referenced in function main TorchProject1 C:\Users\jonah\source\repos\TorchProject1\example-app.obj 1

gcuendet commented 2 years ago

Hi @jonahclarsen !

I added simple CMakeLists.txt to the examples (amongst which the int8/ptq example you mention in this issue) on my branch on #1058 . Compiling and installing the lib using CMake on windows I am able to compile the ptq example without linking error. The CMake configuration command is slightly verbose but here it is:

cmake -S. -Bbuild_dir -DTorch_DIR="..\..\..\..\libtorch\share\cmake\Torch" -Dtorchtrt_DIR="C:\Program Files (x86)\Torch-TensorRT\lib\cmake\torchtrt" -DTensorRT_ROOT="C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-8.2.4.2"

This specifies:

Torch_DIR: since libtorch is not installed system wide.
torchtrt_DIR: it is the install folder you specified when compiling the lib (you can use -DCMAKE_INSTALL_PREFIX to specify a directory, otherwise it looks like it is installed in C:\Program Files (x86) by default.
TensorRT_ROOT: you know that one 😃

Can you try that out and let me know if that solves your linking issue?

jonahclarsen commented 2 years ago

@gcuendet Awesome - that fixed the linking error! Thanks so much.

I'm now getting a "CUDA illegal memory access" error with my implementation of the ptq example after a few batches are calibrated. I'm not sure if this is due to my implementation or if it's a problem with the libraries.

EDIT: It was a problem with my implementation. I was using the wrong input shape. Works now!

pytorch / TensorRT