Questions about using data from real vehicles for testing

[wk565]

Hello！Is it possible to use data from real vehicles and do migration development using pre-trained models? The data would contain the following information as mentioned in the paper: forward looking RGB images, roadmap, speed, actions performed by the expert, BeV semantic segmentation (8 semantic categories: background, road, lane markings, vehicles, pedestrians, and traffic light status)

[anthonyhu]

I expect pre-trained models from the CARLA simulator to not generalise well on real data. There might be a positive transfer, but it will most probably not work without fine-tuning on real data.

[wk565]

However, after I uploaded the map in opendrive format, the error shows that I don't have the corresponding h5 map, can I get the h5 map based on the opendrive map? In addition, I found that the main function in the test code contains a lot of carla-gym environment settings, so how do I use the pre-trained model to test the real car data, do I need to create a new gym environment by myself? Looking forward to your reply!

[xiaoxiguaa]

However, after I uploaded the map in opendrive format, the error shows that I don't have the corresponding h5 map, can I get the h5 map based on the opendrive map? In addition, I found that the main function in the test code contains a lot of carla-gym environment settings, so how do I use the pre-trained model to test the real car data, do I need to create a new gym environment by myself? Looking forward to your reply!

Thank you for your reply! I want to ask you, how long did you run the generated video? How long does it take to evaluate?

[anthonyhu]

If you want to test the model in a closed-loop setting with a real-world simulator, in that case yes you'd need to create a new gym environment. If you want to run the model offline in open-loop, you should only need the RGB images as input and the model should be able to predict future actions and bird's-eye view states.

From what I recall, the longest I've run the generation was for 120seconds. Evaluation takes a long time.. it depends on your hardware but on a RTX 3090 it takes 2+ days

[wk565]

Thanks for your reply, now I want to do open loop testing, do you have code in your project about open loop testing？ Or do I need to write code to load the model, post-process and visualize the results？

[anthonyhu]

You'll need to write that yourself, the code included in the codebase is only for closed-loop testing :)

[wk565]

Okay, thank you very much for your reply！