search

daniel89710 / trt-depth-anything

TensorRT depth-anything for anyone and anywhere
13 stars 2 forks source link

Internal Error (Network has dynamic or shape inputs, but no optimization profile has been defined.) #1

Open robin-ai-ml opened 5 months ago

robin-ai-ml commented 5 months ago

hi I want to run the depth-anythin vits on Nvidia Orin NX board, I migrated some code from tensorrt8.6 to tensorrt 10.0 when I run ./build/trt-depth-anything --onnx depth_anything_vits14.onnx --precision fp32 --batch_size 1 --height 518 --width 518 i always got errors: Error Code 4: Internal Error (Network has dynamic or shape inputs, but no optimization profile has been defined.) can you help me? details are:

**Configuration: HW: NVDIAI ORIN NX 16GB Board SW: CUDA Version: 12.4.131 Cudnn version:9.1.0 TensorRT: 10.0.0.6

****Modified code:**** migrated some code from tensorrt 8.6 to 10.0

setMaxBatchSize is deprecate in tensort 10.0.06 version.

if (batchconfig.at(0) > 1 && (batchconfig.at(0) == batchconfig.at(2))) { // Attention : below API is deprecated in TRT8.4 // Line modified: setMaxBatchSize deprecated, handling with dynamic shapes // builder->setMaxBatchSize(batchconfig.at(2)); // Assuming 'inputTensorName' needs to be replaced with actual input tensor names auto profile = builder->createOptimizationProfile(); profile->setDimensions( network->getInput(0)->getName(), nvinfer1::OptProfileSelector::kMIN, nvinfer1::Dims4{batchconfig.at(0), 3, 128, 128}); profile->setDimensions( network->getInput(0)->getName(), nvinfer1::OptProfileSelector::kOPT, nvinfer1::Dims4{batchconfig.at(1), 3, 518, 518}); profile->setDimensions( network->getInput(0)->getName(), nvinfer1::OptProfileSelector::kMAX, nvinfer1::Dims4{batchconfig.at(2), 3, 640, 960}); config->addOptimizationProfile(profile);

} else { if (buildconfig->profile_per_layer) { auto profile = builder->createOptimizationProfile(); profile->setDimensions( network->getInput(0)->getName(), nvinfer1::OptProfileSelector::kMIN, nvinfer1::Dims4{batchconfig.at(0), input_channel, input_height, input_width}); profile->setDimensions( network->getInput(0)->getName(), nvinfer1::OptProfileSelector::kOPT, nvinfer1::Dims4{batchconfig.at(1), input_channel, input_height, input_width}); profile->setDimensions( network->getInput(0)->getName(), nvinfer1::OptProfileSelector::kMAX, nvinfer1::Dims4{batchconfig.at(2), input_channel, input_height, input_width}); config->addOptimizationProfile(profile); } }

nvinfer1::Dims TrtCommon::getBindingDimensions(const int32_t index) const { //#if (NV_TENSORRT_MAJOR 1000) + (NV_TENSORRT_MINOR 100) + (NV_TENSORPATCH * 10) >= 8500 auto const & name = engine->getIOTensorName(index); auto dims = context_->getTensorShape(name); bool const has_runtime_dim = std::any_of(dims.d, dims.d + dims.nbDims, [](int32_t dim) { return dim == -1; });

if (has_runtimedim) { return dims; } else { return dims; } //#else // auto const & name = engine->getIOTensorName(index); // auto dims = context_->getTensorShape(name); // return dims; //#endif }

bool TrtCommon::setBindingDimensions(const int32t index, const nvinfer1::Dims & dimensions) const { auto const & name = engine->getIOTensorName(index); return context_->setInputShape(name, dimensions); }

****execute log information:**** :~/trt-depth-anything$ ./build/trt-depth-anything --onnx depth_anything_vits14.onnx --precision fp32 --batch_size 1 --height 518 --width 518 [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 19, GPU 4855 (MiB) [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +1156, GPU +1195, now: CPU 1311, GPU 6188 (MiB) [I] [TRT] ---------------------------------------------------------------- [I] [TRT] Input filename: depth_anything_vits14.onnx [I] [TRT] ONNX IR version: 0.0.8 [I] [TRT] Opset version: 17 [I] [TRT] Producer name: pytorch [I] [TRT] Producer version: 2.1.2 [I] [TRT] Domain:
[I] [TRT] Model version: 0 [I] [TRT] Doc string:
[I] [TRT] ---------------------------------------------------------------- L10 [conv 14x14 (1) /14] -1x-1x3 -> -1x-1x384 weights:225792 GFLOPs:0 L124 [resize] L1546 [conv 1x1 (1) /1] -1x-1x384 -> -1x-1x48 weights:18432 GFLOPs:3.6864e-05 L1563 [conv 1x1 (1) /1] -1x-1x384 -> -1x-1x96 weights:36864 GFLOPs:7.3728e-05 L1580 [conv 1x1 (1) /1] -1x-1x384 -> -1x-1x192 weights:73728 GFLOPs:0.000147456 L1596 [conv 1x1 (1) /1] -1x-1x384 -> -1x-1x384 weights:147456 GFLOPs:0.000294912 L1597 [conv 3x3 (1) /2] -1x-1x384 -> -1x-1x384 weights:1327104 GFLOPs:0 L1598 [conv 3x3 (1) /1] -1x-1x48 -> -1x-1x64 weights:27648 GFLOPs:5.5296e-05 L1599 [conv 3x3 (1) /1] -1x-1x96 -> -1x-1x64 weights:55296 GFLOPs:0.000110592 L1600 [conv 3x3 (1) /1] -1x-1x192 -> -1x-1x64 weights:110592 GFLOPs:0.000221184 L1601 [conv 3x3 (1) /1] -1x-1x384 -> -1x-1x64 weights:221184 GFLOPs:0.000442368 L1611 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1613 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1649 [resize] L1650 [conv 1x1 (1) /1] -1x-1x64 -> -1x-1x64 weights:4096 GFLOPs:8.192e-06 L1660 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1662 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1666 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1668 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1704 [resize] L1705 [conv 1x1 (1) /1] -1x-1x64 -> -1x-1x64 weights:4096 GFLOPs:8.192e-06 L1715 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1717 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1721 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1723 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1759 [resize] L1760 [conv 1x1 (1) /1] -1x-1x64 -> -1x-1x64 weights:4096 GFLOPs:8.192e-06 L1762 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1764 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1768 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1770 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x64 weights:36864 GFLOPs:7.3728e-05 L1772 [resize] L1773 [conv 1x1 (1) /1] -1x-1x64 -> -1x-1x64 weights:4096 GFLOPs:8.192e-06 L1774 [conv 3x3 (1) /1] -1x-1x64 -> -1x-1x32 weights:18432 GFLOPs:3.6864e-05 L1813 [resize] L1814 [conv 3x3 (1) /1] -1x-1x32 -> -1x-1x32 weights:9216 GFLOPs:1.8432e-05 L1816 [conv 1x1 (1) /1] -1x-1x32 -> -1x-1x1 weights:32 GFLOPs:6.4e-08 L1852 [resize] Total 0.00250272 GFLOPs Total 2.80426 M params Building... "depth_anything_vits14.fp32-batch1.engine" [I] [TRT] Start build engine [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 370, GPU 6198 (MiB) [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +1154, GPU +229, now: CPU 1524, GPU 6427 (MiB) [I] [TRT] ---------------------------------------------------------------- [I] [TRT] Input filename: depth_anything_vits14.onnx [I] [TRT] ONNX IR version: 0.0.8 [I] [TRT] Opset version: 17 [I] [TRT] Producer name: pytorch [I] [TRT] Producer version: 2.1.2 [I] [TRT] Domain:
[I] [TRT] Model version: 0 [I] [TRT] Doc string:
[I] [TRT] ---------------------------------------------------------------- [E] [TRT] 4: [network.cpp::validate::3521] Error Code 4: Internal Error (Network has dynamic or shape inputs, but no optimization profile has been defined.) [E] [TRT] Fail to create host memory [I] [TRT] End build engine Segmentation fault (core dumped)

daniel89710 commented 4 months ago

Currently, dynamic shapes are not supported in this repository. If you change the ONNX shape to static, it should work.