Gathering Depth(Meter) from ZED SDK and real-time operation

[wkyoun]

I have sucessfully run zed-opencv programm on NVIDIA TK1 board.

First question

However, I want to find the exact depth information of specific pixel.

when I click the location of pixel while running zed-opencv programm, It seems to me that it produce the depth(meter) of pixel that I click

The related function is as follows:

static void onMouseCallback(int32_t event, int32_t x, int32_t y, int32_t flag, void * param) 

float* ptr_image_num = (float*) ((int8_t*) data->data + y_int * data->step); 
float dist = ptr_image_num[x_int] / 1000.f; 

I don't understand why the variable "dist" in this functions is the depth information(METER) of pixel?

In summary, How Can I extract the specific depth information of specific pixel?

Another question

In README file, it states as follows:

Warning :
•  GPU->CPU readback is time-consuming
•  This sample is not designed to operate in real time

It seems to me that this programm is not designed to operate in real time

Then, How can I make the programm operate in real time?

Is there any example for me to follow?

[obraun-sl]

Hi,

1) To extract the depth information for a specific pixel, you need to call first sl::zed::Mat depthMat = zed->retrieveMeasure(DEPTH) and get the value of the pixel (i,j)

float* depthBuffer = (float*)depthMat.buffer;
float depth_in_mm = depthBuffer[i+j*depthMat.step];

the depth is given in mm by default in the ZED SDK. To get it in meters, you can either divide by 1000.f the value you just retrieved (like in the sample) or call zed->setBaselineRatio(0.001) before the grab function (it will convert the depth in meter).

2) By using the TK1, the best would be to run your processing on the GPU as much as possible. the ZED SDK provides GPU output that can be used directly in CUDA or OpenGL. You can take a look at the gpu samples for that. If you need to have cpu processing, then the best is to use zero copy technics (on TK1/TX1) to share cpu/gpu buffers. Either use cudaMallocManaged() or cudaHostAlloc()/CudaHostGetDevicePointer() to do so.

Best, /OB/

[wkyoun]

@sl-braun thank you for kind comments!

I still have some questions: Now I understand that I need to divide the value by 1000 to get it in meters, as follows:

float dist = ptr_image_num[x_int] / 1000.f; 

But, What I don't understand is as following sentence:

float* ptr_image_num = (float*) ((int8_t*) data->data + y_int * data->step);

based on your explanation,

above, variable "ptr_image_num" is itself the value of the pixel in mm. Is that right?

I cannot understand how to get the value of the pixel in order to obtain the depth of the pixel exactly. (Actually, I am not familiar to opencv)

what is the meaning of data->data + y_int * data->step? (I understand that i+jdepthMat.step in your code is the index of pixel as follows)

Mat image(ROW, COL, CV_TYPE);
DATA_TYPE* data = (DATA_TYPE*)image.data;
data[WANT_ROW *  image.cols + WANT_COL]
- WANT_ROW : accessing row
- WANT_COL :  accessing column

but It looks like that

data->data + y_int * data->step is the index of the pixel, but doesn't make sense to me.

Would you please explain why data->data + y_int * data->step is the index of the pixel?

For example, I cannot understand "data->data + y_int * data->step"

More Specifically,

In following link:

https://www.stereolabs.com/blog/index.php/2015/07/14/tutorial-4-save-zed-depth-maps-for-matlab-purpose/

This is the code to save depth information as matlab file(.dat) In following code,

std::cout << "Saving depth >>";
std::ofstream depthFile;
depthFile.open("depth.dat");
float* ptr_d;
for (int i = 0; i < depth.height; ++i)
{
   ptr_d = (float*)(depth.data + i * depth.step);
   for (int j = 0; j < depth.width * depth.channels; ++j)
   {
      depthFile << ptr_d[j];
      // if not end of the current row, we add a space character
      if (j != (depth.width * depth.channels) - 1)
      depthFile << " ";
   }

depthFile << "\n"; } depthFile.close(); std::cout << " DONE" << std::endl;

I cannot understand more specifically ptr_d = (float)(depth.data + i * depth.step); for (int j = 0; j < depth.width \ depth.channels; ++j) { depthFile << ptr_d[j];

Would anyone of you explain why ptr_d is defined as this way? It looks like a really simple problem related to matrix handling in opencv, but doesn't make sense

Thank you in advance

[adujardin]

Hi, Just to clarify, accessing a pixel like this is the "C way" but it's equivalent to depthMat.at<float>(y, x) if you prefer.

[wkyoun]

I finally found the five ways of accessing pixel in opencv in following link:

http://longstryder.com/2014/07/which-way-of-accessing-pixels-in-opencv-is-the-fastest/

it looks like that

the way of accessing pixel in ZED-OpenCV is

OpenCV iteration method 3 – raw pointer access with raw step : 166014 (usecs) : 100 % of min

Would any one of you give me a opinion?

Thank you in advance!

[wkyoun]

@sl-braun Thank you for your help I have some questions:

There are two functions to downloads the measure (disparity, depth or confidence of disparity) from the device as followings:

run image processing on the GPU means using retrieveMeasure_gpu function instead of retrieveMeasure? (in the same ways : using normalizeMeasure_gpu instead of normalizeMeasure?)

Mat sl::zed::Camera::retrieveMeasure 
Mat sl::zed::Camera::retrieveMeasure_gpu 

Mat sl::zed::Camera::normalizeMeasure 
Mat sl::zed::Camera::normalizeMeasure_gpu 

[adujardin]

If speed is your concern, the fastest way is to use the raw pointer on a line, the same way the sample or the link you provided ("method 3") does. Behind the scene, the line is often put in cache so accessing each pixel is very fast.

Concerning the GPU, yes to retrieve a GPU buffer you have to use retrieveMeasure_gpu.

[Patelvishal95]

Hey i guess this does not work float* depthBuffer = (float*)depthMat.buffer;

you have to use float* depthBuffer = (float*)depthMat.data;

[vishnukumarts]

Hi,

To extract the depth information for a specific pixel, you need to call first sl::zed::Mat depthMat = zed->retrieveMeasure(DEPTH) and get the value of the pixel (i,j)

float* depthBuffer = (float*)depthMat.buffer;
float depth_in_mm = depthBuffer[i+j*depthMat.step];

the depth is given in mm by default in the ZED SDK. To get it in meters, you can either divide by 1000.f the value you just retrieved (like in the sample) or call zed->setBaselineRatio(0.001) before the grab function (it will convert the depth in meter).

By using the TK1, the best would be to run your processing on the GPU as much as possible. the ZED SDK provides GPU output that can be used directly in CUDA or OpenGL. You can take a look at the gpu samples for that. If you need to have cpu processing, then the best is to use zero copy technics (on TK1/TX1) to share cpu/gpu buffers. Either use cudaMallocManaged() or cudaHostAlloc()/CudaHostGetDevicePointer() to do so.

Best, /OB/

I tried to use cudaMallocManaged() but i am facing issues. I am using my own depth algorithm, requesting frames from zed mini camera.

I created two gpumat using unified memory concept(CudaMallocManaged) of CUDA. When i try call convertTo function it got crashed, with below error,

what(): OpenCV(3.4.2) /home/visionteam/revampedtarastack/source/build/opencv-3.4.2/modules/cudev/include/opencv2/cudev/grid/detail/transform.hpp:321: error: (-217:Gpu API call) an illegal memory access was encountered in function 'call'

I tried in ubuntu18.04 with cuda 10

` Mat input = imread("sample.jpg",0);

uint8_t* h_a;

uint8_t* h_b;

cudaMallocManaged(&h_a, sizeof(uint8_t)1280720);

cudaMallocManaged(&h_b, sizeof(float)1280720);

cv::Mat left1(input.cols,input.rows,CV_8UC1,h_a);

h_a = (uint8_t*)input.data;

cv::cuda::GpuMat floatGPU(input.rows,input.cols,CV_32FC1,h_b);

cv::cuda::GpuMat leftGPU(input.rows,input.cols,CV_8UC1,h_a);

floatGPU.convertTo(leftGPU,CV_8U);

When i execute i am receiving this error. what(): OpenCV(3.4.2) /home/visionteam/revampedtarastack/source/build/opencv-3.4.2/modules/cudev/include/opencv2/cudev/grid/detail/transform.hpp:321: error: (-217:Gpu API call) an illegal memory access was encountered in function 'call'