Carla Sensor Manipulation

[StevenCoral]

Hi, I know this had been asked and not answered before, but perhaps there is a better way of introducing the problem.

I would like to create new sensors (and share the contribution), or edit existing ones, or adding post-processing materials for the cameras (such as fisheye lenses). I have gone through the source files but I got lost at some point. I've read numerous posts talking about registering them and so on, or about ongoing development (such as here or here , but its just not enough to go through the process from start to end. Assuming I have some proficiency with UE4, I would like to have a series of actions needed to be done in order to enable the spawning of a my custom sensor via the Python API in the same manner as the other sensors are spawned. For example:

• Create a new C++ class
• Include header: a.h, b.h, c.h
• Call a registering function
• Do the sensor magic you need to do
• Call the serialization function once your data is packed A short description to each step, focused on the Carla structure orientation would be also helpful. Unpacking the data in python is pretty straightforward, I just want to be able to stream data out of UE4 using the Carla interface. Usage of post-processing materials for camera is a bonus (distortions can also be generated in other ways).

I really hope the developers can help with this, if a short tutorial like so is out of the question. Heck, once I get it right, I'll write this tutorial myself.

Many thanks, and really looking forward to getting a lead on this, Steve

[nsubiron]

I did a pull request exactly with that :smile: (#1884), still under review though.

Meanwhile you can read it here https://github.com/carla-simulator/carla/blob/nsubiron/sensors-tutorial/Docs/dev/how_to_add_a_new_sensor.md

[StevenCoral]

That's amazing, just what I needed. I only went through it briefly but I intend to try and utilize it soon. Are you the point of contact for new sensors? And, I saw in the carla timeline that radar sensors are on the agenda - is there already work in progress? Is there any reason to start making one myself or will it be a double effort? Would they need help from someone who has experience with developing automotive grade radars?

[nsubiron]

I can help if you have issues with the current system, but I'm no longer working in the Carla team so I'm not up-to-date on how's the implementation of the radar progressing. The idea is to build a sort of parallel simulation for radar and lidar sensors to simulate them realistically (intensity depending on material, reflection, etc.), but that will take time. If for you works having a simplified version (like the current lidar implementation), then I'd say might be worth it to implement it yourself.

[StevenCoral]

Well, there are many ways this could be implemented, I don't see a reason to elaborate too much about pros and cons, thatll be long. There were even times where I thought I'll use the lidar, but there is no option to change the section (sometimes you don't want a 360-degree view, velodyne allows you to choose a section within it). I did make something preliminary using a depth camera in python but I need more flexibility, hence the "add new sensor" thing. Got any idea who I should turn to just to make sure I'm not wasting time?

[StevenCoral]

@nsubiron First of all, many thanks for the tutorial which I see now is in the readthedocs page. I have created my sensor. It works fine on the Unreal side by UE_Logs (drawdebughelpers gave me a dll linkage error for some reason - they do work on custom UE4 projects). I can also find it in the blueprints library and check out it's attributes. But, upon the callback function to my_sensor.listen, I get a carla.lincarla.SensorData class instance that has only the frame, frame_number, timestamp and transform members (and they also get updated with data). But I dont have anything that contains my actual data (for ease of use, I made my data structure an Array type. just a long string of floats. But, as said, they don't appear anywhere under the callback argument, its not an iterable like in your example but a SensorData object.

I would like to discuss whats wrong but it might be a little long with the files and all, is there a more convenient way to talk about it?

Steve

[xmyqsh]

@StevenCoral If your problem is about the process of adding a new sensor, eg. how to custom the sensor data structure with UE4 type system, how to design the sensor data and serialize and deserialize it efficiently. All the things related to adding the sensor into carla, I can help you. But, for the process of sensor data generation, I cannot help too much.

You could ping me at # development channel (https://discord.gg/rxTWd9f)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.