search

Unity-Technologies / Unity-Robotics-Hub

Central repository for tools, tutorials, resources, and documentation for robotics simulation in Unity.
Other
2.06k stars 425 forks source link

Unity Rgb-d Simulated Camera to ROS #345

Open panagelak opened 2 years ago

panagelak commented 2 years ago

Hello i really like this project!

I believe the Community miss an easy example/functionality of simulating an rgbd camera in Unity and transferring the topics to Ros efficiently.

I have created a starter project for this purpose here : https://github.com/panagelak/rgbd_unity_camera

I believe the best way to do that is to send into ros 4 topics namely

  1. a compress rgb image and camera info topic (which i have successfully done)
  2. a compress depth image and depth camera info topic ( which i need help with!!)

Then we can decompress the rgb and depth images into ros and combine them with the camera info topics to construct colored or not pointclouds, through the image transport packages.

For convenience i have included this packages from the ros image_transport packages, which i have modified slightly in order to include them as libraries.

I tried to do it by myself but unfortunately i am not very familiar with Unity and shaders : (

I also included some code from here : https://www.immersivelimit.com/tutorials/unity-depth-camera-simulation But the depth image i managed to get was a gray image of rgb format instead of the 32FC1 that ros expects in depth images.

So i think the difficult part will be to do the following two things

  1. Create an example of a unity depth camera that uses a shader with 32FC1 Format.
  2. Create an Encode (compress) algorithm that compresses this format, like the data = texture2D.EncodeToJPG(qualityLevel)

If somebody can help me with that, that would be great!

You can make me a pull request on my repo for now (ros-noetic, Unity 2020.3.25), in order to not have to meet high standards but just a working code and later i think it would be a great addition to have as an example (along with ros actions P)

Thank you, I am awaiting your input

unity_image ros_image

github-actions[bot] commented 2 years ago

This issue has been marked stale because it has been open for 14 days with no activity. Please remove the stale label or comment on this issue, or the issue will be automatically closed in the next 14 days.

vidurvij-Unity commented 2 years ago

Thank you posting this. I will look into this issue.

panagelak commented 2 years ago

@vidurvij-Unity

Thank you for taking the time

ZhuoyunZhong commented 2 years ago

Do you have the problem of frame drop when you publish images to ROS and show them in Rviz? My framerate can drop from ~200Hz to 40Hz when publishing images.

florianfhws commented 2 years ago

Dear @vidurvij-Unity i am working on this problem as well, for me the problem is that the publishing and the rendering of the cameras is done in the same function and therefor my depth camera to wait till the color camera is finished. I know that rendering is bound to one thread so stacking an infinite amount of cameras is not possible. However is it possible to put the publishing of the images in a separate thread that is called from the function that renders the camera image? So for clarification I have two cameras on each of the cameras is a script with a InvokeRepeating function in these function the camera image gets rendered and then put into a ros message and then transmitted. Lets call the functions cam1func and cam2func. So the problem is cam2func has to wait till cam1func is fully procesed to start even so the rendering part of cam1func which blocks the mainthread is done cam2func will not start.

florianfhws commented 2 years ago

Right now another problem is that packing the image data into a message is taking up a lot of time (20-30 ms) by EncodeToJPG() as the data should be compressed to avoid huge data to be transferred. can this process also be packed into a thread?

panagelak commented 2 years ago

@florianfhws

you might be interested in the implementation at https://github.com/fsstudio-team/ZeroSimROSUnity (there they don't compress it and i had to run the rosbridge from the provided docker after i fixed some minor build errors)

also i raised an issue with a problem on the distortion of the pointcloud (due to what i believe incorrect depth data representation and possibly cam info)

https://github.com/fsstudio-team/ZeroSimROSUnity/issues/20

also the format of the depth image should be 32fc1 and i'm not sure if you can use EncodeToJPG()

florianfhws commented 2 years ago

@panagelak the project is very interesting especially the use of AsyncGPUReadbackRequest. However as the request only enables access to the raw data you would first need to copy them into a Texture2D and then call the encodetojpg function and again encodetojpg would cause the mainthread to wait slow down (15 ms), so packing raw data into your messages is the only fast option. A solution with packing the raw data into a opencv matrix and then converting it also slowed down the callback of AsyncGPUReadbackRequest too much (3-4 ms). Both values when wanting to render and therefor calling AsyncGPUReadbackRequest every 0.05s