search

CarkusL / CenterPoint

Export CenterPoint PonintPillars ONNX Model For TensorRT
MIT License
204 stars 50 forks source link

engine.cpp (1036) - Cuda Error in executeInternal: 700 (an illegal memory access was encountered) #11

Closed OrcunCanDeniz closed 2 years ago

OrcunCanDeniz commented 2 years ago

Created the onnx models without any problem. There was also nothing wrong with compilation of TRT sample. However I get the below output when sample is run.

filePath[idx]: ../data/centerpoint//points/0106a9b8e65f4ad1867b44591aeed8b0.bin [12/30/2021-15:56:31] [I] [INFO] pointNum : 282378 [12/30/2021-15:56:31] [I] PreProcess Time: 31.4462 ms [12/30/2021-15:56:31] [I] inferenceDuration Time: 9.84006 ms [12/30/2021-15:56:31] [I] PostProcessDuration Time: 1.9788 ms filePath[idx]: ../data/centerpoint//points/048a45dd2cf54aa5808d8ccc85731d44.bin [12/30/2021-15:56:31] [I] [INFO] pointNum : 278690 [12/30/2021-15:56:31] [I] PreProcess Time: 30.7371 ms [12/30/2021-15:56:31] [I] inferenceDuration Time: 9.28456 ms [12/30/2021-15:56:31] [I] PostProcessDuration Time: 11.9031 ms filePath[idx]: ../data/centerpoint//points/05bc09f952ab4cf8b754405f20d503c5.bin [12/30/2021-15:56:31] [I] [INFO] pointNum : 243328 [12/30/2021-15:56:31] [I] PreProcess Time: 22.8271 ms [12/30/2021-15:56:31] [I] inferenceDuration Time: 10.7552 ms [12/30/2021-15:56:31] [I] PostProcessDuration Time: 2.65588 ms filePath[idx]: ../data/centerpoint//points/06be0e3b665c44fa8d17d9f4770bdf9c.bin [12/30/2021-15:56:31] [I] [INFO] pointNum : 258553 [12/30/2021-15:56:31] [I] PreProcess Time: 29.4288 ms [12/30/2021-15:56:31] [I] inferenceDuration Time: 9.25186 ms [12/30/2021-15:56:31] [I] PostProcessDuration Time: 7.31504 ms filePath[idx]: ../data/centerpoint//points/07fad91090c746ccaa1b2bdb55329e20.bin [12/30/2021-15:56:31] [I] [INFO] pointNum : 285130 [12/30/2021-15:56:31] [I] PreProcess Time: 29.1292 ms [12/30/2021-15:56:31] [I] inferenceDuration Time: 9.2483 ms [12/30/2021-15:56:31] [I] PostProcessDuration Time: 8.86581 ms filePath[idx]: ../data/centerpoint//points/092051710b7b4d9294e98dcb3f0f7be1.bin [12/30/2021-15:56:31] [I] [INFO] pointNum : 285204 [12/30/2021-15:56:31] [E] [TRT] engine.cpp (1036) - Cuda Error in executeInternal: 700 (an illegal memory access was encountered) [12/30/2021-15:56:31] [E] [TRT] FAILED_EXECUTION: std::exception [12/30/2021-15:56:31] [E] [TRT] engine.cpp (169) - Cuda Error in ~ExecutionContext: 700 (an illegal memory access was encountered) [12/30/2021-15:56:31] [E] [TRT] INTERNAL_ERROR: std::exception [12/30/2021-15:56:31] [E] [TRT] Parameter check failed at: safeContext.cpp::terminateCommonContext::216, condition: cudnnDestroy(context.cudnn) failure. [12/30/2021-15:56:31] [E] [TRT] Parameter check failed at: safeContext.cpp::terminateCommonContext::221, condition: cudaEventDestroy(context.start) failure. [12/30/2021-15:56:31] [E] [TRT] Parameter check failed at: safeContext.cpp::terminateCommonContext::226, condition: cudaEventDestroy(context.stop) failure. [12/30/2021-15:56:31] [E] [TRT] ../rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered) terminate called after throwing an instance of 'nvinfer1::CudaError' what(): std::exception

OrcunCanDeniz commented 2 years ago

I was running the sample under /usr/src/tensorrt/samples. Then I moved samples/ under ~/ the error stopped occuring. However now I'm facing free(): invalid next size (normal). I observed that infer() function ends succesfully, however any of the part coming after the execution of the infer() cant execute.

zhangtingyu11 commented 2 years ago

I was running the sample under /usr/src/tensorrt/samples. Then I moved samples/ under ~/ the error stopped occuring. However now I'm facing free(): invalid next size (normal). I observed that infer() function ends succesfully, however any of the part coming after the execution of the infer() cant execute.

Have you solved the problem?I got the same error.

OrcunCanDeniz commented 2 years ago

I was running the sample under /usr/src/tensorrt/samples. Then I moved samples/ under ~/ the error stopped occuring. However now I'm facing free(): invalid next size (normal). I observed that infer() function ends succesfully, however any of the part coming after the execution of the infer() cant execute.

Have you solved the problem?I got the same error.

No but it seems all pcd files are inferred. Do you think so?

OrcunCanDeniz commented 2 years ago

As the main problem is different than the one that made me open this issue I will now close this one and open another #12 .