my vehicle remains stationary all the time

[zygalaxy]

When I run the pre-training model, my vehicle remains stationary all the time. How can I solve it?

[Kait0]

Which branch are you using? On the 2021 branch this is expected (should not happen always but can happen in multiple routes). This is an instance of causal confusion/ Inertia Problem and the model in the 2021 is strongly affected by it. The newer models in the 2022 branch are better and have this problems less (but still do). The 2022 branch additionally has a creeping rule to posthoc fix this problem. We noticed that causal confusion is not robust and can easily be broken if the vehicle slowly roles forward, which we do if the car is stuck for a long time (e.g.: see Table 10 in the paper).

[zygalaxy]

Thank you for your quick and accurate answer. I use the 2022 version. The vehicle remains stationary all the time, only occasionally moving a little to the right and front. Can this be solved?

[zygalaxy]

I tried: 1) using -vulkan to solve this problem, but I can only use -oepngl, but I can't use -vulkan. Once I use -vulkan, carla0.9.10.1 can't display the picture. 2) I also tried -quality-level=Low, but the vehicle never moves.

[Kait0]

opengl is fine. This sounds more like a problem with the setup then. My first guess is that the model weights aren't loaded correctly (this can fail silently and use uninitialized weights). Thy debugging this here.

[zygalaxy]

Thank you for your patience in answering. I debugged the areas you pointed out. Through temp = net.load_state_dict(state_dict, strict=False), 1) For transfuser, I got temp = _IncompatibleKeys(missing_keys=[], unexpected_keys=['_model.lidar_encoder._model.stem.conv.weight']). 2) For late_fusion, I got temp=<All keys matched successfully>, does this mean that the weights have been loaded correctly, but in both cases, the vehicle remains stationary! I'm very distressed about this issue.

[Kait0]

These should be fine, these keys are mismatched because the models were trained with an older codebase. Can you try the visualization function and send an example here? You will know that the model weights are loaded correctly if those look reasonable.

You need to set the environment variable SAVE_PATH=/path/to/log/folder as well as put the debug variable to True in the config. to use the visualization function.

[zygalaxy]

The global position of the vehicle is shown in Figure 1. The result of visualization is occasionally Figure 2 and Figure 3, but the position of the vehicle remains unchanged all the time.

It seems that it is because the white block in the middle affects the vehicle's progress.

[Kait0]

Hm that is very strange. It means the central camera is failing occasionally. This should not happen. I have never seen this before, you need to debug to see why this is happening. In general how the camera image is created, is by having 3 high resolution cameras, cropping out the center part (to remove image distortions) and then concatenating them together.

It is expected that the vehicle doesn't move if the vehicle breaks every couple frames. You need to throttle a lot of frames consecutively in CARLA for the car to start driving.

The model itself seems to be loaded correctly.

[zygalaxy]

I found that the white block in the middle was only an accidental phenomenon. I collected the visualization results twice again and found no white block in the middle, but the vehicle never moved!

[Kait0]

The white bar is not just because of the visualization as it also shows up in the model prediction (e.g. depth map), but perhaps its rare?

In the example you showed above the stuck detector is true, so the vehicle should be forced to move forward if the code is working correctly. Does the model itself predict that it should move? Did you test different routes?

[zygalaxy]

Thank you for your patient answer. I will try again.