/sonar_image is black

[zym19981209]

I always obtain full black images on the /blueview_p900/sonar_image topic. When I use “rostopic echo /blueview_p900/sonar_image_raw”, the returned intensity data is all zero. How can we solve it ?

[woensug-choi]

Sorry for what's happend! Can you explain how you ran it? Which environment and which commands?

[zym19981209]

I'm on Ubuntu 20.04 LTS with ROS1 Noetic. I use the following command: roslaunch nps_uw_multibeam_sonar sonar_tank_blueview_p900_nps_multibeam.launch. When I operate on a laptop with NVIDIA GEFORCE GTX 950m graphics card, I always obtain full black images on the /blueview_p900/sonar_image topic. But when I execute the same command on a desktop computer with NVIDIA GeForce RTX 2070 SUPER graphics card, I can subscribe to the topic normally. Is this problem caused by the difference of computer graphics card performance?

[woensug-choi]

Oh! I was hard time reproducing the issue problem. Thank you very much for trying it out on a different machine! Can you check which NVIDIA driver version and CUDA version you are testing with? try nvidia-smi command to see the details.

[zym19981209]

The laptop with NVIDIA GEFORCE GTX 950m graphics card, NVIDIA Driver Version: 510.60.02, CUDA Version: 11.6. The desktop computer with NVIDIA GeForce RTX 2070 SUPER graphics card, NVIDIA Driver Version: 510.54, CUDA Version: 11.6.

[woensug-choi]

Hmm.... could not reproduce the black image. Could you possibly try using docker environment at your laptop?

[woensug-choi]

I am also suspecting insufficient GPU memory. It may require at least 8 GB. This could be tested by reducing the number of total rays(not raySkips, but vertival FOV)

[estellearrc]

Why the vertical FOV and not raySkips? What is this parameter raySkips supposed to do? I was about to increase its value because I only have 2GB GPU memory when I saw your comment.

[woensug-choi]

@estellearrc The raySkips is to skipping the rows when doing the ray calculation. Skipping from already built matrix data. In other words, the horizontal/vertical FOV defines the size of the matrix that will be allocated to memory. A higher value for raySkips would not reduce the size of the matrix.

Thank you again! for tying it out!

[estellearrc]

Unfortunately, changing the values of these parameters didn't solve the black sonar image problem... But I presume my 2GB-size GPU memory is really limiting because I get an "Out of memory" error on a CUDAMalloc using the cuda-memcheck  tool. I've tried the plugin on different hardware architecture (3GB and 4GB memory), there's a huge actualization latency on the sonar images (5 to 20 seconds) but they are not black.

[estellearrc]

By the way, where have you found the bandwidth parameter for each sonar? I saw it is linked to the sonar image size which is nFreq nBeams and not nRays nBeams. Why do you use nFreq rather than nRays for the sonar image's height?

[woensug-choi]

sonar image's height is related to range. And the range is related to the number of datasets (number of frequencies before ifft). Rays are extended into not forward direction but downward(depth) direction. The final image is conceptually a three-dimensional cloud point projected into a two-dimensional image which eliminates the downward direction (depth direction; ray panning direction).

[woensug-choi]

@estellearrc About the huge latency, are you using a ray-based or raster-based version? Also, you may want to reduce range and increase raySkips. Also, if you are close to the ground (grazing the floor) and capture lots of cloud points (number of objects intersected with rays), it will decrease the refresh rate. https://field-robotics-lab.github.io/dave.doc/contents/dave_sensors/Multibeam-Forward-Looking-Sonar/#benchmarks

[estellearrc]

Ok thanks a lot for your answers! I'm using the raster-based version with the Oculus m1200d sonar model with a NVIDIA GEFORCE RTX 3060 (12GB GPU-memory), I don't get any black image anymore. Nevertheless, when I isolate the CUDA code for sonar calculation to generate only one sonar image, I still get an "Out of memory" error on a CUDAMalloc using cuda-gdb. Do you have any idea why?

[woensug-choi]

Hmmm... I've used the function cudaMemGetInfo to check the GPU memory usage for roslaunch nps_uw_multibeam_sonar sonar_tank_blueview_p900_nps_multibeam.launch

      // CUDA Memory usage
      int num_gpus;
      size_t free, total;
      double used_gb, free_gb, total_gb;
      cudaGetDeviceCount( &num_gpus );
      for ( int gpu_id = 0; gpu_id < num_gpus; gpu_id++ ) {
          cudaSetDevice( gpu_id );
          int id;
          cudaGetDevice( &id );
          cudaMemGetInfo( &free, &total );
          total_gb = ( (double)total) / 1e9;
          free_gb = ( (double)free) / 1e9;
          used_gb = total_gb - free_gb;
          printf("GPU (id = %d) memory: used = %0.2f GB, free= %0.2f GB, total= %0.2f GB\n",
                id, used_gb, free_gb, total_gb);

I got the results something like this. And it did not accumulate as time passes.

GPU 0 memory: used = 1.11 GB, free= 5.34 GB, total= 6.44 G

Nothing close to out-of-memory warnings. I can't imagine how it reports out-of-memory while showing the image. You can check the memory usage on the fly with nvidia-smi during the run. Does your 12 GB almost max up during the run?

[estellearrc]

Thank you for checking it out! Indeed that's unexpected to have non-black images but GPU out of memory error... What do you get with roslaunch nps_uw_multibeam_sonar sonar_tank_oculus_m1200d_nps_multibeam.launch? During the run, the 12GB GPU memory doesn't max up at all (around 1.5GB used) but the GPU-util compute memory does reach 100% sometimes. And in terms of image quality, I do not obtain as good results as those shown in the sonar wiki. It's like there's an image folding on the right-hand side.

[estellearrc]

When I use the ray version, I get something like this

[woensug-choi]

@estellearrc The first image you posted is what you would expect for the raster version. Just as an image on the left side of those local search scenario examples on the wiki document. However, the second image doesn't look like anything that I am familiar with. What I get from roslaunch nps_uw_multibeam_sonar local_search_blueview_p900_nps_multibeam_ray.launch look like

To make the image look exactly like the wiki document, use the MATLAB script at `scripts/plotRawData_GPURay_based.m

P.S. the results I get from roslaunch nps_uw_multibeam_sonar sonar_tank_oculus_m1200d_nps_multibeam.launch shows like these images. I can't get those GPU memory errors on my machine.

[estellearrc]

Thanks a lot for all your explanations and advice, I've finally found why the ray version wasn't working. I was using the wrong velodyne_simulator (not the official one). Everything is working fine now! By the way, I have migrated your sonar plugin to ROS2 (galactic). If you're interested, I can open a PR on a new branch dedicated to ROS2?

[estellearrc]

I am also suspecting insufficient GPU memory. It may require at least 8 GB. This could be tested by reducing the number of total rays(not raySkips, but vertival FOV)

To close this issue, the reason why the sonar images were black was indeed a shortage of GPU-util compute memory.

[woensug-choi]

I am also suspecting insufficient GPU memory. It may require at least 8 GB. This could be tested by reducing the number of total rays(not raySkips, but vertival FOV)

To close this issue, the reason why the sonar images were black was indeed a shortage of GPU-util compute memory.

Closing it as we found out the reason.