search

gazebosim / gazebo-classic

Gazebo classic. For the latest version, see https://github.com/gazebosim/gz-sim
http://classic.gazebosim.org/
Other
1.15k stars 478 forks source link

Simulating Thermal Cameras #1192

Open osrf-migration opened 10 years ago

osrf-migration commented 10 years ago

Original report (archived issue) by Stefan Kohlbrecher (Bitbucket: Stefan_Kohlbrecher).

Every now and then, requests for simulating different spectra than the default visible light pop up, notably thermal imaging. I implemented a quick hack for simulating thermal cameras as described here on ROS Answers and available in hector_gazebo_thermal_camera, but that clearly only works for some scenarios. This ticket is for discussion and tracking any effort towards simulating thermal cameras. I do not have the time to work on this, but maybe someone else steps in.

Issue #550 about arbitrary model properties is related to this one as implementing those could be a proper path towards thermal imaging simulation.

osrf-migration commented 10 years ago

Original comment by Unknown Name (Bitbucket: casro).

I suggest the next milestone: a non-photo-realistic simulator resembling this one: http://youtu.be/OFwMbJ3UosQ?t=22s

Any ideas?

osrf-migration commented 10 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

This would be a nice feature. I think the best approach would involve creating a new Camera type in gazebo that uses a different glsl shader.

osrf-migration commented 10 years ago

Original comment by Isura (Bitbucket: isura).

Our research group will be working in this direction. Arbitrary model properties may be a good idea or a standardization of a thermal property. @Ahsan_Habib

osrf-migration commented 10 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

Great to hear. Let us know if you need any help and guidance.

osrf-migration commented 9 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

osrf-migration commented 7 years ago

Original comment by David Rajaratnam (Bitbucket: daveraja).

Just wondering if anyone has made progress on this?

I am particularly interested in the idea of arbitrary model properties (more so than a simulated thermal camera) and for my purposes want a virtual (depth) camera that assigns a unique color to each model.

If no-one has made progress on this then I'll try to hack something myself, but I'm not at all confident about how far I'll get. I have limited time, not much experience with the gazebo source code, and most importantly zero experience with 3D rendering, Ogre, glsl shaders, etc,... so any hints would be appreciated.

osrf-migration commented 7 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

osrf-migration commented 5 years ago

Original comment by Andrew Price (Bitbucket: arprice).

@daveraja, did you ever have any success with rendering each model as a unique color?

osrf-migration commented 5 years ago

Original comment by David Rajaratnam (Bitbucket: daveraja).

@arprice. I can't recall exactly what I did (it's been a while and I've moved away from robotics for the time being) but I did manage to get something working but only through some pretty horrible hacks.

I never managed to understand Ogre and shaders enough that I could implement things properly so I ended up maintaining a global table that assigned a unique colour to individual models when they are inserted in gazebo. I think I actually replaced the material of the object with that solid colour (but only in the gazebo server and not in the gui).

Ultimately, it worked for what I needed but I don't think it is anything that would be useful to anyone else.

osrf-migration commented 5 years ago

Original comment by Andrew Price (Bitbucket: arprice).

That's funny, those are the two approaches I'm considering. I think having custom shaders is the right way to go, but I'm struggling with how to pass a variable down to that level.

osrf-migration commented 5 years ago

Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).

but I'm struggling with how to pass a variable down to that level.

if you're looking for an example on how to pass variable to shaders, here's one. You'll need to declare that variable in the materials script and in the shaders

osrf-migration commented 5 years ago

Original comment by Andrew Price (Bitbucket: arprice).

Thanks @iche033, those look like exactly what I need! I think I get the broad strokes of how to add a label/segmentation camera, but there's a lot of layers in the shader->material->compositor->sensor->plugin relay race to understand.

osrf-migration commented 5 years ago

Original comment by Andrew Price (Bitbucket: arprice).

After a bit of investigation, I'm still wandering in the wilderness of all the various layers of Gazebo/Ogre. As alluded to earlier, my interest is specifically in publishing a "segmentation image" along the lines of the depth image functionality, where each pixel value is the id of the model at that pixel. I've successfully created and loaded a plugin with a custom material that takes a parameter named id and writes that as the color, but there's a number of other steps that I'm unclear on. For instance:

@iche033 or @nkoenig, can either of y'all advise on a path forward? I'm happy to write the feature as far as I am able, but my knowledge of gazebo's inner workings is currently pretty weak. If this would be straightforward to implement for someone with more framework familiarity but is currently a low priority, maybe there's a way to compensate them for the time investment required? From a logistics perspective, I'm happy to create a different issue/pull request/plugin repo as necessary.

osrf-migration commented 5 years ago

Original comment by David Rajaratnam (Bitbucket: daveraja).

Hi @arprice. I am not an expert with Gazebo/Ogre and also spent a lot of time in the wilderness of the various layers. So I don't know if this will help but my modified gazebo (https://bitbucket.org/daveraja/gazebo-old) might give you some ideas.

Note: It's out of date, it's pretty messy, and I probably can't really explain exactly why I did certain things (it was a while ago)! Also I wouldn't try to compile it as I made lots of hacks to the cmake files to be able to build and run it side by side with the standard gazebo versions (my requirements at the time).

However, the main commit that might be useful is https://bitbucket.org/daveraja/gazebo-old/commits/54005ab2743535d1044d5991b9ea5176df92f8f6

In particular, I created a new sensor ObjectCamera by basically duplicating DepthCamera; so, adding a gazebo::rendering::ObjectCamera and gazebo::sensor::ObjectCameraSensor etc.

The code that replaces the model's material with a custom colour is in gazebo::rendering::ObjectUtil.[hh|cc]. I had to modify quite a few other files to make all this work, in particular to call my wrapper create_entity function where ever the original code called the OgreSceneManager::createEntity() function.

Hope this helps in some way.

osrf-migration commented 5 years ago

Original comment by Andrew Price (Bitbucket: arprice).

Thanks @daveraja, I'll take a look at it!