Differences between the real image and the simulation image

[cabinx]

The image sonar simulator is really an excellent work. I found some differences between the simulation image and the real image. The situation is similar to Cirqueira’s work(https://github.com/romulogcerqueira/sonar_simulation).

The two images below is from A MULTI-DEVICE SONAR SIMULATOR FOR REAL-TIME UNDERWATER APPLICATIONS . The first image is the acoustic data captured by Tritech Gemini 720i sonar equipped on the FlatFish AUV, and the next is generated by the simulator.

---------------------------------------------real image------------------------------------------------------------ -----------------------------------------simulation image--------------------------------------------------------

Let’s focus on the seafloor. In polar coordinate, I think the seafloor in the image should be arc instead of straight as the seafloor is flat. We got arcs (the seafloor in the images) in our test with a Tritech 1200ik image sonar too.

I know very little about the acoustic modeling. The reason leads to this situation, I guess, may be in the step of computing distance between the sonar and the target point. The simulator gets the distance with a depth camera. But I think the depth may not be the distance r. For example, a sketchin in the horizontal plane. The parameters refer to the _sonar_calculationcuda.cu, in the function _sonarcalculation. The depth camera provide the depth data instead of distance data. The same as in the vertical plane.

Considering the horizontal and the vertical, I think the distance between the target point in the depth image should be:

So I tried to modify the code in _sonar_calculationcuda.cu, the function _sonarcalculation: distance = distance * sqrt(1 + pow(tan(ray_azimuthAngle), 2.0) + pow(tan(ray_elevationAngle), 2.0));

But It doesn’t work, the image generated is vary strange. Then I tried: distance = distance / sqrt(1 + pow(tan(ray_azimuthAngle), 2.0) + pow(tan(ray_elevationAngle), 2.0));

I get an amazing result. In the test scence, sonar pose is [0 0 5 0 0.759382 1.524432]. Some sonar parameters, such is the horizontal_fov, are also changed. ---------------------------------------------------------test scence---------------------------------------------------------- + + + -------------------------------------------------------before modified------------------------------------------------------ + + + -------------------------------------------------------after modified------------------------------------------------------

Now I confused whether my work is right, since it’s not in accordance with the formula derivation.

If anyone can solve my confusion, I will be very appreciate.

[woensug-choi]

@cabinx Sorry for the late response! I have just noticed this issue is here.

First of all, thank you for trying the plugin out! Really appreciate end-users like you who looked into the codes!

The confusion here is that what you see in the results of this plugin (on your plot with your colour bar scale) are showing objects captured in the scene. Yes, the depth camera can only capture the scene projected into a rasterization plane (this means that the scene data input is clipped with a square projection window with straight lines). But it is distance, not depth.

Let me explain with Blueview P900 positioned at the same location as your case (You may try this by launching roslaunch nps_uw_multibeam_sonar sonar_tank_blueview_p900_nps_multibeam.launch as it is on this repo and remove sonar tank and cylinders in Gazebo by mouse and reposition the sonar using the left panel inside the Gazebo).

The results I get look like this, (replotted at MATLAB script https://github.com/Field-Robotics-Lab/nps_uw_multibeam_sonar/blob/main/scripts/plotRawData.m). The vertical stripes at X from 6 to 8 are the bottom object being captured. The launch file included in the repo is set to use only about 11 elevation rays [footnote 1]. Your scene has no gaps, which mean you have very fine elevation rays without any skips.

Those stripes are the object in the scene being captured. And you see calculated sonar data (including reverberation) does show the arc at X from 4 to 6. In the real sonar data, the bottom reflection is very low and has relatively similar data to reverberation in the scene.

I am sure you have more questions. My next goal is to make an auto-adjusting resolution for elevation rays. And there seem to be some issues that scene is clipped out if the objects are too close. It's all related.

Happy to continue the discussion :)

[footnote 1]

Total 114 elevation rays https://github.com/Field-Robotics-Lab/nps_uw_multibeam_sonar/blob/dbb0c7e6ddfab6d3af8e72a791db7936b6a0a2f8/models/blueview_p900_nps_multibeam/model.sdf#L43 but uses only every 10 rays for a faster refresh rate. https://github.com/Field-Robotics-Lab/nps_uw_multibeam_sonar/blob/dbb0c7e6ddfab6d3af8e72a791db7936b6a0a2f8/models/blueview_p900_nps_multibeam/model.sdf#L67 This means that if you decrease the raySkips you will see more vertical stripes at X from 6 to 8.

[cabinx]

@woensug-choi Thank you for your attentive explanation. I shared this question with my team members, well, the discussion ended without a consistent conclusion now. Please allow me to make a brief introduction to my work with the simulator. My work is not in the acoustics simulation area.

We built an underwater observation dataset, near the seaside about 10 meters depth, with eight categories of targets, including cube, ball, cylinder, tyre, etc. The dataset was collected with two Tritech 1200ik image sonar and a camera. We provide the image raw data, stored in bmp format, in the dataset. Annotations were also made for target recognition and classification, mainly in AI areas. We organized an online underwater target recognition and classification competition with the dataset in the first-half of this year. We plan to opensource this dataset after further collation. We develop a annotation tool for the image sonar raw data.

During the simulation, I record the raw data in a rosbag. The frames data were transform to bmp format after simulation. Bellow is an example(a ball) of contradistinction of real image and simulation image in the tool. Attention, the image is in cartesian coordinates. The horizon is azimuth angle info and the vertical is range info.

If I don't modified the codes, the simulation image would be:

The work of collecting underwater data is really very very hard, complex and costly. Therefore, underwater opensource data resources are really scarce. Actually, our dataset is too samll for many research. The lack of data limit many underwater researchers. I think the simulation is a solution to the problem. Thanks to the development of image translation technology in AI areas, we can make the simulation image close to the reality. Just like this.

I am doing some work with the image from the dataset and the sonar simulator, using GAN(Generative Adversarial Networks). After the training, we can get a generator, which can translate the simulation image to the reality image. I hope the researchers can generate the data they want only sitting in front of the computer in the future.

If my work get good results, we plan to provide the simulation data and the reality data at the same time in next competition. The competitors need to consider image translation and target recognition and classification at the same time to get a better score. We will require the winning competitors to opensource their works. We hope our work can make some contribution to the development of underwater research.

Here I want to seek the permission to use your simulator in the competition next year. Of course, your work will be added to the contributors of the dataset then.

Thank you again for your significant work and I looking forward to your future work.

[woensug-choi]

@cabinx Thank you for sharing your dataset examples. We are very much interested in working with AI fields where this plugin development could be most useful! I've been reading about papers on the subject to tackle the idea to utilize the plugin to fasten the AI learning process. I don't think this simulated dataset can or should replace the real data but definitely can benefit in providing data for the 0th iteration.

Looking at the results you have shared, I have a hunch that how you mapped the dataset into cartesian could differ from what we thought. Have you tried using the MATLAB script provided in the repo? https://github.com/Field-Robotics-Lab/nps_uw_multibeam_sonar/blob/main/scripts/plotRawData.m

The code block where it calculates the azimuthal spaces in polar coordinates are fixed to this form due to generic depth camera plugin configurations this plugin is based on. https://github.com/Field-Robotics-Lab/nps_uw_multibeam_sonar/blob/dbb0c7e6ddfab6d3af8e72a791db7936b6a0a2f8/scripts/plotRawData.m#L35-L39

Also, I've noticed that the how the actual sonar discretizes the azimuth beam displacements are hardware-specific, Issue https://github.com/Field-Robotics-Lab/nps_uw_multibeam_sonar/issues/13. Which I have not taken into consideration yet.

I would love to collaborate with your research subject and provide what you need. I am also searching for research topics on the AI subject from my side too. Could you provide me with your dataset in detail so that I can try things out?

By the way, what raySkips value did you use for your image?

[woensug-choi]

I've generated a more analogous results to compare. This is generated out of the most recent commit (The latest commit now exports the image to rqt_image_viewer as float values. You may change the colormap on the rqt_image_viewer (in default, it's grey-coloured). I've found the Pink most fancy.) Left : Real, Middle: Cirquira's, Right: This plugin. The arc (red box) is speckle noise produced by the object in the scene and can be seen on both left and right images. The straight line is the bottom captured from the scene. The only limitation of this plugin is the resolution in ray(vertical) that end up distinguishable lines for the seabed on the right image.

[cabinx]

@woensug-choi Now I know I make a big mistake. We are discussing in different coordinates. The seafloor is arc in the polar coordinate, and that's my perspective. When transformed to cartesian coordinate, as the work of the script "plotRawData.m", the seafloor is streight. It's another perspective, as shown below. Forgive my poor sketch. Yes, the simulation is correct.

Can I make a little suggestion? Maybe it will be more friendly to the developers if an example is provided on how to use script "plotRawData.m". It took some time to derivate the format of "SonarRawData_000001.csv" in the script. Yeah, the script "bag2file.py" provide the data of every frame. But the script "plotRawData.m" needs data with the format of "SonarRawData_000001". Anyway, I successfully run the script and awared my mistake. Here's the result.

The dataset's info refers to the following link: https://code.ihub.org.cn/projects/14186

The dataset is for "2021 CHINA UNDERWATER ROBOT PROFESSIONAL CONTEST"-"UNDERWATER OBJECT DETECTION ALGORITHM CONTEST"-"ACOUSTIC OBJECT DETECTION". The contest is supported by PengCheng Laboratory, ShenZhen, China.

The dataset is not the final edition. In fact, the quality of the dataset is not very well since we tried this kind of work for the first time. We collected the data in a hurry considering of the deadline and many works were not well prepared. We are not satisfied with this dataset and will collect new data months later.

A characteristic of our dataset is that the sonar raw data is provided for for target recognition and classification. The intensities matrix is stored as a bmp file.

Thanks again for your excellent work! I will close this issue in a week. I will be pleased for further discussion if there's any questions, and my email is kaibng@mail.ustc.edu.cn .

[woensug-choi]

@cabinx Glad to hear that you got it working!

I get '请输入提取码' msg when trying to download the dataset. Does the dataset include descriptions for the environments too? If I can, I would like to make a close comparison with the dataset for the wiki page. If you can help me with that, it would be great!

I've updated the wiki to for the matlab script.

[cabinx]

@woensug-choi Ah, I was not careful enough while reading the wiki, now I know I should set the "writeLog" flag on. The password of the dataset, "05dg", is behind the download link.

The dataset only include some sonar information in the annotation files. I didn't participate in the data collectting work. But my workmate, who is the data collecter, is writing a working document. I can ask he to share some environment information on the dataset webside too, mainly about the depth fo the seafloor, the size of the targets, etc. I can also email a brief copy about the environment information to you when he finish if you need. If there are any other questions, you can also contact the dataset maintainer. His email is in the Data Description on the dataset webside.

The dataset is opensource, so it's welcome to use it on the underwater research. But we hope that the users can add a brief citation as below:

The dataset is made by PengCheng Laboratory, ShenZhen, China.

Currently, we are on the initial stage of making an opensource underwater acoustic dataset. The quality of the dataset is obviously need to be improved. So the idea of making a close comparison will most likely be very difficult to get a satisfying result. But we will provide help as much as we could.

[woensug-choi]

@cabinx Pretty impossible to get access to. When I tried to download, it required a downloader to be installed into the system. I did. Then, it requires a Baidu ID to use the downloader. Which I don't have. Making an account seems to require a Chinese phone number which I do no have. Could you try hosting it on a google drive or something? It had three zip files, quick explanations?

[cabinx]

@woensug-choi Sorry! Now I share the dataset on onedrive. https://1drv.ms/u/s!AlpdPhejQ4FqiDL25YVhjxxT1WeU?e=DcwBRD

[woensug-choi]

@cabinx Do you happen to have any progress on the comparisons? comparison between the experiment and the simulated result.

[cabinx]

@woensug-choi Actually，we make a simple experiment of object recognition with CycleGan.

400 real images, which with only a cube target, are selected from the dataset mentioned before. We use 300 of the images as the trainning set, and the rest as the test set. Then we create 300 simulaition images, with only a cube target too. Next, we handle the simulation images with CycleGan to realize the image translation. At last we make a comparison experiment with different models trainned from different dataset. The result is showed below.

Besides, the mAP is about 50 if only simulation image is used as the trainning set.

The result shows that the simulated data works in neural networks. And after image translation from the simulation area to the real area, we can get a better effection. We make some plans for future research. But a new research task of object reconstruction occupied most of our time. Maybe we will continue our plans after the task.

My colleague has finished an article(under review) mainly about a optimized network. He use the part of the work above. We select the article showed in the wiki of the project dave as the reference: "Physics-Based Modelling and Simulation of Multibeam Echosounder Perception for Autonomous Underwater Manipulation".

Thank you again for your excellent work! Hope we can keep in touch so that we can get your help for the future research.

[woensug-choi]

@cabinx OMG! It's amazing to hear that it can be useful! I personally am very interested in piping in the simulated image to fast-forward the machine learning process. This concept has been discussed more than several times within my team. We once had a chance to discuss this with a doctoral student at Michigan University. But have not heard from him for quite some time now.

Since I am a post-doc with high liberty to research, I would love to engage with your future research. Any articles from the previous works?

[cabinx]

@woensug-choi Our research just started, so we don't have too much achievement. But I'm glad to introduce some recently articles, mainly about simulated FLS images associate with deep learning, by other researchers. These articles inspied us a lot.

"CycleGAN-based realistic image dataset generation for forward-looking sonar" "A Review on Deep Learning-Based Approaches for Automatic Sonar Target Recognition" "Realistic Sonar Image Simulation Using Generative Adversarial Network" "Realistic Sonar Image Simulation Using Deep Learning for Underwater Object Detection" "Sonar-based yaw estimation of target object using shape prediction on viewing angle variation with neural network" "Underwater sonar image classification using generative adversarial network and convolutional neural network"

[woensug-choi]

@cabinx Did your paper (under review) earlier got published?

[kimkimyoung]

I am @cabinx 's teammate. @woensug-choi Thank you for your remarkable work. Since we are new in the field, there aren't too many works yet. We tested and compared the result by fusing real and simulated data performed on a modified network, which is based on YOLO and self-attention mechanism. The link below is the accepted paper but not yet published. https://camps.aptaracorp.com/ACM_PMS/PMS/ACM/ICRAI2021/1/5802219d-5e2e-11ec-b613-166a08e17233/OUT/icrai2021-1.html

[bsb808]

This is very cool. Thank you @kimkimyoung and @cabinx for letting us know about your great work.

[woensug-choi]

@cabinx Very cool indeed! Thank you for sharing the article! I am studying ML myself but not quite yet to fully understand what you've done! I would love to engage more and develop further for ML research.