QDQ translator error on TRT 8.5

[marcoslucianops]

I'm getting Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered error when using TensorRT 8.5.3.1.

[03/28/2024-14:47:50] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 825, GPU 1296 (MiB)
[03/28/2024-14:47:50] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +11, GPU +10, now: CPU 836, GPU 1306 (MiB)
[03/28/2024-14:47:50] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:58] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:47:59] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:00] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x00000000000003ea due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception an illegal memory access was encountered
[03/28/2024-14:48:01] [E] Error[1]: [virtualMemoryBuffer.cpp::~StdVirtualMemoryBufferImpl::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/28/2024-14:48:01] [E] Error[2]: [optimizer.cpp::computeCosts::3988] Error Code 2: Internal Error (Impossible to reformat.)
[03/28/2024-14:48:01] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[03/28/2024-14:48:01] [E] Engine could not be created from network
[03/28/2024-14:48:01] [E] Building engine failed
[03/28/2024-14:48:01] [E] Failed to create engine from model or file.
[03/28/2024-14:48:01] [E] Engine set up failed

It works well with TensorRT 8.6. Is it possible to fix it for the TRT 8.5?

[nvoliver]

Hi @marcoslucianops, this appears to be an issue with TensorRT and not with QDQ Translator. Feel free to raise the same question at https://github.com/NVIDIA/TensorRT/issues.

Closing.