Planning module failed to update reference line after rerouting

[xiafawu]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04): Linux Ubuntu 18.04
• Apollo installed from (source or binary): Source
• Apollo version (3.5, 5.0, 5.5, 6.0): 5.5

Steps to reproduce the issue:

I would like to test sim control (mainly the planning module) by playing the record file containing messages of type apollo.perception.PerceptionObstacles, apollo.perception.TrafficLightDetection, and apollo.prediction.PredictionObstacles. Before playing the record file, I already enabled sim control, the planning module, the perception module, and the routing module. However, there is no guarantee that the planning module can always produce trajectories. The planning module can successfully produce trajectories initially most of the time, but it will start producing messages containing the error "Failed to updated reference line after rerouting" after several iterations, each sends a routing request and then plays the record file. Also, once this error occurred, the planning module will continue to produce messages with the same error for the subsequent iterations.

• enable sim control
• enable the planning, routing, and perception module
• repeat the following steps:
  • send routing request
  • play the record file that only contains apollo.perception.PerceptionObstacles, apollo.perception.TrafficLightDetection, and apollo.prediction.PredictionObstacles messages using cyber_recorder play

Any help or recommendation is much appreciated! Thanks.

Supporting materials (screenshots, command lines, code/script snippets):

• Planning Log The planning log contains the following messages repetitively E0918 04:41:33.150034 3764 reference_line_provider.cc:208] Fail to get reference line E0918 04:41:33.210606 3766 reference_line_provider.cc:543] Failed to extract segments from routing E0918 04:41:33.210633 3766 reference_line_provider.cc:585] Failed to create reference line from routing

[ntutangyun]

Does restarting the planning module helps ?

[jmtao]

@ntutangyun not sure how you ran into this error if there's no rerouting. Once it happens, you'll have to restart planning module (and resend routing for the planning module) to get over it. If you want to know the root cause whey you ran into this error, we'll have to look further on your set up.

[ntutangyun]

@ntutangyun not sure how you ran into this error if there's no rerouting. Once it happens, you'll have to restart planning module (and resend routing for the planning module) to get over it. If you want to know the root cause whey you ran into this error, we'll have to look further on your set up.

Hi, @jmtao i saw this error previous also. I believe it might be caused by the spawn location of the apollo vehicle. If it's not in the centre of the traffic lane, sometimes it would produce this error and the planning module would no longer work until i manually restart it..

I did a little hack modification of the planning module and solved this issue. Whenever it fails to find reference line, it will automatically hard reset everyting and calls the init function again. So it's like automatic restart. 😂

[ntutangyun]

[xiafawu]

@ntutangyun Thanks for the feedback! I tried restarting the planning module by running bash /apollo/scripts/planning.sh stop and then bash /apollo/scripts/planning.sh, which did not resolve the issue. I could only exit the container and run dev_start.sh and dev_into.sh again to make the planning module functional. I think I will try the hard reset approach you provided and post a follow-up comment if it works later on. Thank you!

[xiafawu]

@ntutangyun Sorry for the late response. I tried your approach of implementing a hard reset, and based on the current behavior I think this issue is resolved. Thanks for your help! 😄

[jmtao]

@ntutangyun thanks for trying this out. But, we design it this way on purpose. The planning module has to be restarted manually when it fails to create routing. It's considered a serious failure, and we hope to restart the module to get a clean and safe state, instead of automatically restarting from a bad state, which could be unsafe.

[AkshayDesaiSDCE]

@ntutangyun with this approach, the planner stays active (meaning it does not get stuck in the "failed to update reference line after rerouting") and tries to replan, but I see the planner status flicker between this error and "failed to create reference line" error when the vehicle is on the route. But when I give the route again, the planner plans successfully. Do you know what could be the reason for this?

[ntutangyun]

@AkshayDesaiSDCE I'm not sure, but it may be due to the routing message being obsolete and the planning module requiring a new one to start over.

[AkshayDesaiSDCE]

@ntutangyun so once the route is sent to the planner, with your implementation of the planner reset, does the planner not look for the same route every step? Or are there conditions under which the route is cleared in the code and needs to be resent manually? I basically want the planner to keep trying to replan to the route sent to it initially.

With your code addition the planner stays alive which is certainly a good improvement. However, once the vehicle moves away and comes back closer to the route, the planner fails to snap onto the route again.