search

stereolabs / zed-yolo

3D Object detection using Yolo and the ZED in Python and C++
https://www.stereolabs.com/
MIT License
159 stars 68 forks source link

I'm running into a cuda error: illegal memory access when running zed-yolo on RTX 2080Ti. I didn't have this problem with 1080. Is there anything I need to change in my code. I've posted a part of it and the accompanying error message. #23

Closed saratrajput closed 2 years ago

saratrajput commented 4 years ago

Error

in sl::ERROR_CODE sl::Mat::setFrom(const sl::Mat&, sl::COPY_TYPE, cudaStream_t) : cuda error [77]: an illegal memory access was encountered.
in sl::ERROR_CODE sl::Mat::setFrom(const sl::Mat&, sl::COPY_TYPE, cudaStream_t) : cuda error [77]: an illegal memory access was encountered.
CUDA Error: an illegal memory access was encountered
python3: ./src/cuda.c:36: check_error: Assertion `0' failed.
saratrajput commented 4 years ago

I'm basically getting the error at this part:

cam.retrieve_image(mat, sl.VIEW.VIEW_LEFT)
image = mat.get_data()

Do I need to make any changes for a specific GPU?

obraun-sl commented 4 years ago

Hi, If not already done, you probably need to activate this line in the makefile for RTX cards (CC>=70) https://github.com/stereolabs/zed-yolo/blob/master/libdarknet/Makefile#L27 Let me know if that helps.

saratrajput commented 4 years ago

@obraun-sl Hi, thanks for your reply but I had already uncommented that line when making libdarknet. Also by the way, libdarknet doesn't compile for RTX cards if CUDA version is 10.2. It works only for 10.0.

For now, I've been able to solve the issue by setting the Zed's FPS with:

zed.set_camera_fps(60)

before retrieving the image. I don't know if that's the right approach since it has dropped the overall ZED-Yolo's FPS.

saratrajput commented 4 years ago

So my reasoning is, ZED's FPS was too low before. So when we call cam.retrieve_image and cam.retrieve_measure it tries to copy image data or point cloud data from GPU to CPU which doesn't yet exist causing these errors:

in sl::ERROR_CODE sl::Mat::setFrom(const sl::Mat&, sl::COPY_TYPE, cudaStream_t) : cuda error [77]: an illegal memory access was encountered.
in sl::ERROR_CODE sl::Mat::setFrom(const sl::Mat&, sl::COPY_TYPE, cudaStream_t) : cuda error [77]: an illegal memory access was encountered.

I confirmed ZED's low FPS with:

current_fps = cam.get_camera_fps()
print("Zed FPS: {}".format(current_fps))

before retrieving the image. It was 3.0 and then when we call detections = detect(netMain, metaMain, image, thresh) YOLO tries to perform detection on something that doesn't exit either.

CUDA Error: an illegal memory access was encountered
python3: ./src/cuda.c:36: check_error: Assertion `0' failed.

Do you think I'm right in my reasoning? What causes this low FPS? I didn't encounter this error though with GTX 1050 and 1080. And I checked back with those GPUs again and still ZED's FPS is 3.0 but it doesn't cause CUDA illegal memory. I'm running on Ubuntu 18 btw.

adujardin commented 4 years ago

Hi @saratrajput, I tried the default darknet_zed.py on the same configuration (Ubuntu 18, Cuda 10.0, RTX 2080) without any issues. I tried forcing lower framerate to test for possibly empty buffer or other issue but it always worked.

Since the return code of the grab function is checked here the buffer shouldn't be empty, and no thread or queues are used. When using the default tiny-yolov3 model the FPS displayed is around 40 and 25 FPS with yolov3.

By default, on my config, this code displays a fixed 60fps:

current_fps = cam.get_camera_fps()
print("Zed FPS: {}".format(current_fps))

You should make sure the ZED isn't recognized as USB2.0 by opening ZED Explorer for instance and checking the FPS. Did you try with an SVO file?

About Cuda 10.2, unfortunately, you have to match the Cuda version everywhere and since the ZED SDK is currently only available for Cuda 10.0 for Ubuntu 18 you have to use this (including for cuDNN).

saratrajput commented 4 years ago

Hi @adujardin. Thank you for the detailed reply.

To answer your questions:

I have tested with both ZED and ZED-M cameras. I'll proceed to check if this is being caused by ZED SDK. I'm currently testing on ZED SDK 2.8.4. But basically I didn't have any problems with GTX-1080.

Also here is part of my Makefile. Let me know if there's anything wrong with it:

GPU=1
CUDNN=0
CUDNN_HALF=0
OPENCV=1
AVX=0
OPENMP=0
LIBSO=1
ZED_CAMERA=1

# set GPU=1 and CUDNN=1 to speedup on GPU
# set CUDNN_HALF=1 to further speedup 3 x times (Mixed-precision on Tensor Cores) GPU: Volta, Xavier, Turing and higher
# set AVX=1 and OPENMP=1 to speedup on CPU (if error occurs then set AVX=0)

USE_CPP=0
DEBUG=1

#ARCH= -gencode arch=compute_30,code=sm_30 \
#      -gencode arch=compute_35,code=sm_35 \
#      -gencode arch=compute_50,code=[sm_50,compute_50] \
#      -gencode arch=compute_52,code=[sm_52,compute_52] \
#     -gencode arch=compute_61,code=[sm_61,compute_61]

OS := $(shell uname)

# Tesla V100
# ARCH= -gencode arch=compute_70,code=[sm_70,compute_70]

# GeForce RTX 2080 Ti, RTX 2080, RTX 2070, Quadro RTX 8000, Quadro RTX 6000, Quadro RTX 5000, Tesla T4, XNOR Tensor Cores
ARCH= -gencode arch=compute_75,code=[sm_75,compute_75]
adujardin commented 4 years ago

I think you should try with CUDNN=1. You will have to install it (you can check the TF doc). Without cudnn the detection code takes a lot more GPU memory and is much slower.

Do you have issues with the default ZED SDK samples, without darknet?

If this is caused by the ZED SDK it would be useful to get a minimal snippet that could reproduce the issue. To help debug you could use valgrind or cuda-memcheck to track the origin of the error.

saratrajput commented 4 years ago

Yes, I noticed that and re-tried with CUDNN=1, but the same error persists.

I'll try to debug if it's being caused by ZED SDK. Thanks.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days