The results are different with or without a printf function call in cuda kernel ?

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

https://developer.nvidia.com/tensorrt

Apache License 2.0

10.15k stars 2.08k forks source link

The results are different with or without a printf function call in cuda kernel ? #3960

Closed lix19937 closed 1 week ago

lix19937 commented 1 week ago

Description

In trt plugin, when there is a printf call inside cuda kernels, the calc result is diff without printf call in cuda kernel. So strange !

Environment

TensorRT Version:trt8510

NVIDIA GPU:rtx3070

NVIDIA Driver Version:

CUDA Version:11.4

CUDNN Version:11.6

Operating System:ubuntu20.04

lix19937 commented 1 week ago

After multi-check, it is cuda's bug for IO bus is busy. Remove printf calls in kernels, will get right answer/result.