search

open-rmf / rmf_ros2

Internal ROS infrastructure for RMF
Apache License 2.0
74 stars 60 forks source link

Gracefully fail tasks that have become impossible #16

Open mxgrey opened 3 years ago

mxgrey commented 3 years ago

Feature request

Description

There may be situations where a task has been assigned and is being executed, but the robot encounters an insurmountable error (the robots breaks down? an unmovable obstacle blocks its path?). We should have a way to gracefully handle these errors. Currently the fleet adapter will simply crash when a task phase emits an error.

Recommendation:

When a robot encounters a critical error on a task, it should give up and inform the task dispatcher that it cannot complete the task. Then the task dispatcher can open a new bid on the task. If no bidders can perform the task, then the dispatcher should notify the human operators that the task cannot be completed. Ideally that notification should provide as much information as possible about why the task failed.

Achllle commented 3 years ago

+1 on this. Does the following log match this case?

[full_control-14] [SearchForPath] CRITICAL ERROR: Impossible plan requested! Participant [fetch26] owned by [fetch] Requested path (11) --> (8)
[full_control-14] terminate called without an active exception

Sometimes simply trying again is a good strategy. Perhaps that should be a parameter or more elegantly the robot that failed should be able to participate in the next bid again.

mxgrey commented 3 years ago

Yes, the log you're showing there matches the topic of this issue.

I believe the only way you should be able to get this error right now is if you explicitly close the lanes needed by every possible route that the robot can use to reach the goal. Is that what happened here?

Sometimes simply trying again is a good strategy.

Trying again is an option, but lane closures have an indefinite duration, so it might not be clear when it makes sense to try again. Maybe the dispatcher can listen to the lane closure topic and re-issue the request for bids whenever it sees a lane open up.

the robot that failed should be able to participate in the next bid again.

When the bidding happens, every fleet will have the opportunity to bid on behalf of its robots. If the lane(s) have been reopened by the time the bidding has started, then the robot will be able to re-win the bid. Otherwise if the lanes are still closed and the task is still impossible for the robot, it simply won't submit a bid at all.

Achllle commented 3 years ago

I did not manually close any lanes. The robot did lose its localization right before that. I'm not sure if that could cause this issue. Since I'm testing with real robots this might be hard to reproduce reliably.

mxgrey commented 3 years ago

It's plausible that losing the localization could lead to this.

The error indicates that the fleet adapter believes the robot is on or near waypoint #11 and trying to find a path to waypoint #8, but the planner believes that no valid path exists between the two. I realize it's a bit of a chore to identify which waypoints correspond to which indices, but I would suggest checking if the graph contains any valid path from waypoint #11 to waypoint #8. According to the planner, there shouldn't be.

If the planner is correct that #8 is unreachable from #11 then I suppose the localization issue that you mentioned may have caused the fleet adapter to mistakenly believe that the robot was on/near #11 when it wasn't supposed to be, and that could lead to this error happening.

osama-z-salah commented 2 weeks ago

@mxgrey I am wondering how to notify the fleet adapter that my robot had encounters error given that my api can give me this data