search

PJLab-ADG / LimSim

LimSim & LimSim++: Integrated traffic and autonomous driving simulators with (M)LLM support
https://pjlab-adg.github.io/LimSim/
GNU General Public License v3.0
415 stars 35 forks source link

Carla crashed when using new map in limsim++ #35

Open Sephiroth1118 opened 1 month ago

Sephiroth1118 commented 1 month ago

Hi! I'm recently using limsim++'s VLM model. My carla is a build from source version, which is directly running in UE editor. The CarlaTown06 example worked very well. But when I use a map created by myself, the UE editor crashed. It seems the map missed some tiles so that the engine crashed.

The strange thing is the the new map worked very well in Limsim. What I want to ask is that is there any difference when using carla between limsim and limsim++? The limsim++ do not need to run run_synchronization.py. I checked the code in Model.py and saw some differences but did not get the point.

Thank you so much for your time and consideration!

Fdarco commented 1 month ago

Hi, I'm sorry that you encountered this problem when using LimSim++. Since we have never used custom network files on LimSim++, we don't know what caused your simulation engine to crash. However, LimSim++ actually also uses run_synchronization.py, but we have made some modifications to the original file to simplify the user's usage process. There is this code in lines 176-189 of simModel.Model.py. The main changes we made to run_synchronization.py are adding code to get the surround image of ego car from CARLA and adjusting the control logic of TraCI. These changes should not involve the network file. Perhaps you can send a screenshot of the engine crash and the error report, and we can try to help solve it.

Sephiroth1118 commented 1 month ago

Hi, I'm sorry that you encountered this problem when using LimSim++. Since we have never used custom network files on LimSim++, we don't know what caused your simulation engine to crash. However, LimSim++ actually also uses run_synchronization.py, but we have made some modifications to the original file to simplify the user's usage process. There is this code in lines 176-189 of simModel.Model.py. The main changes we made to run_synchronization.py are adding code to get the surround image of ego car from CARLA and adjusting the control logic of TraCI. These changes should not involve the network file. Perhaps you can send a screenshot of the engine crash and the error report, and we can try to help solve it.

Hi! Thank you so much for your reply!

This is the error report of the engine crash, there is only a few words: Assertion failed: Tile [File:/home/Workplace/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/MapGen/LargeMapManager.cpp] [Line: 619]

Actually I think it is an error totally caused by UE editor. I test some other maps. All the maps that have this problem are the maps I created by a Carla Tool named digital twin tool, which can generate 3D maps according to OSM and XODR file. These maps consist of multiple sub-maps, which is considered as large maps. The problem is usually about UE expects some missing data.

This morning I tried a stupid way and solved this poblem. I just move all the objects in sub-map to the main map and then delete the sub-map. Then everything works well. So I guess the problem may be caused by getting data from sub-maps. But I'm not sure. I think it is the way to get the surrounding image of ego car leads to the problem. May I ask how do you realize it?

This problem is not very related to limsim and limsim++, so just treated it as an interesting sharing. Although I did not fix the bug, I solved the problem in another way :).

Fdarco commented 1 month ago

I'm glad you found a solution to the problem. I think your statement is correct, because when getting the surround image, you need to use the map to project it on the camera's view. LimSim does not need this projection. So your custom map can run normally on LimSim. If you want to delete the code for getting the surround image, you can find them in sumo_integration.run_synchronization.py. Mainly in the two classes SimulationSynchronization and CAMActor.

Sephiroth1118 commented 1 month ago

I'm glad you found a solution to the problem. I think your statement is correct, because when getting the surround image, you need to use the map to project it on the camera's view. LimSim does not need this projection. So your custom map can run normally on LimSim. If you want to delete the code for getting the surround image, you can find them in sumo_integration.run_synchronization.py. Mainly in the two classes SimulationSynchronization and CAMActor.

Thank you for your reply! I will check the code! By the way, I saw there are 6 camera settings including 3 back cameras. So actually we can give 6 images to VLM?

Fdarco commented 1 month ago

Yes, LimSim++ provides a 6-view surrounding images, which is the same as nuscenes camera settings. However, for the sake of a more aesthetically pleasing layout of the GUI, we only show 3 front-view images. The model.getCARLAImages() function returns a List[CameraImages]. You can find the exact definition in class CameraImages in simModel.DataQueue.py.

Sephiroth1118 commented 1 month ago

Yes, LimSim++ provides a 6-view surrounding images, which is the same as nuscenes camera settings. However, for the sake of a more aesthetically pleasing layout of the GUI, we only show 3 front-view images. The model.getCARLAImages() function returns a List[CameraImages]. You can find the exact definition in class CameraImages in simModel.DataQueue.py.

Thank you so much for your reply!