Errors about using c# and TensorRT

[PTA00]

I tried cuda12 and 11, and the corresponding cudnn version, and all attempts failed. I guarantee that the path is correct and everything is fine when using only dml and cuda, but wrong when using tensorRT. Before that, I looked at all the relevant issues, which took me 6 hours. I was wondering if anyone had successfully generated a program using c# and TensorRT. On .NET8.0 (c#+cuda+cudnn+tensorRT+windows-x64)

I think my head is going to explode.

Unhandled exception. Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:RuntimeException] 
D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1209 onnxruntime::ProviderLibrary::Get 
[ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load 
"C:\Users\PTA00\source\repos\ConsoleApp1\ConsoleApp1\bin\Debug\net8.0\runtimes\win-x64\native\
onnxruntime_providers_tensorrt.dll"

[PTA00]

Here's the test code, from the example: I only made a slight modification,Switch to tensorRT.

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.ML.OnnxRuntime.Tensors;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;

namespace Microsoft.ML.OnnxRuntime.ResNet50v2Sample
{
    class Program
    {
        public static void Main(string[] args)
        {
            // Read paths
            string modelFilePath = "resnet50-v2-7.onnx";
            string imageFilePath = "dog.jpeg";

            // Read image
            using Image<Rgb24> image = Image.Load<Rgb24>(imageFilePath);

            // Resize image
            image.Mutate(x =>
            {
                x.Resize(new ResizeOptions
                {
                    Size = new Size(224, 224),
                    Mode = ResizeMode.Crop
                });
            });

            // Preprocess image
            Tensor<float> input = new DenseTensor<float>(new[] { 1, 3, 224, 224 });
            var mean = new[] { 0.485f, 0.456f, 0.406f };
            var stddev = new[] { 0.229f, 0.224f, 0.225f };
            image.ProcessPixelRows(accessor =>
            {
                for (int y = 0; y < accessor.Height; y++)
                {
                    Span<Rgb24> pixelSpan = accessor.GetRowSpan(y);
                    for (int x = 0; x < accessor.Width; x++)
                    {
                        input[0, 0, y, x] = ((pixelSpan[x].R / 255f) - mean[0]) / stddev[0];
                        input[0, 1, y, x] = ((pixelSpan[x].G / 255f) - mean[1]) / stddev[1];
                        input[0, 2, y, x] = ((pixelSpan[x].B / 255f) - mean[2]) / stddev[2];
                    }
                }
            });

            // Setup inputs
            var inputs = new List<NamedOnnxValue>
            {
                NamedOnnxValue.CreateFromTensor("data", input)
            };
//change--------------------------------

            using var gpuSessionOptions = SessionOptions.MakeSessionOptionWithTensorrtProvider(0);
            using var session = new InferenceSession(modelFilePath, gpuSessionOptions);

            // Run inference
            //using var session = new InferenceSession(modelFilePath);
//change---------------------------------

            using IDisposableReadOnlyCollection<DisposableNamedOnnxValue> results = session.Run(inputs);

            // Postprocess to get softmax vector
            IEnumerable<float> output = results.First().AsEnumerable<float>();
            float sum = output.Sum(x => (float)Math.Exp(x));
            IEnumerable<float> softmax = output.Select(x => (float)Math.Exp(x) / sum);

            // Extract top 10 predicted classes
            IEnumerable<Prediction> top10 = softmax.Select((x, i) => new Prediction { Label = LabelMap.Labels[i], Confidence = x })
                               .OrderByDescending(x => x.Confidence)
                               .Take(10);

            // Print results to console
            Console.WriteLine("Top 10 predictions for ResNet50 v2...");
            Console.WriteLine("--------------------------------------------------------------");
            foreach (var t in top10)
            {
                Console.WriteLine($"Label: {t.Label}, Confidence: {t.Confidence}");
            }
        }
    }
}

[PTA00]

Microsoft.ML.OnnxRuntime.OnnxRuntimeException:“[ErrorCode:RuntimeException] 
D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1209 
onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 ""
 when trying to load "C:\Users\PTA00\Desktop\sample\Microsoft.ML.OnnxRuntime.ResNet50v2Sample
\bin\Debug\net8.0\runtimes\win-x64\native\onnxruntime_providers_tensorrt.dll"

[yuslepukhin]

This is what I would do. dumpbin /DEPENDENTS onnxruntime_providers_tensorrt.dll" and then try running where on each of them. I don't think dependency walker still works well.

[yuslepukhin]

BTW, I recommend using OrtValue API for direct memory access. This reduces amount of garbage.