Race condition when updating goal for ExePath

[RainerKuemmerle]

The following is run in a thread

//AbstractActionBase
  virtual void run(ConcurrencySlot &slot)
  {
    slot.execution->preRun();
    runImpl(slot.goal_handle, *slot.execution);
    ROS_DEBUG_STREAM_NAMED(name_, "Finished action \"" << name_ << "\" run method, waiting for execution thread to finish.");
    slot.execution->join();
    ROS_DEBUG_STREAM_NAMED(name_, "Execution completed with goal status "
                           << (int)slot.goal_handle.getGoalStatus().status << ": "<< slot.goal_handle.getGoalStatus().text);
    slot.execution->postRun();
    slot.in_use = false;
  }

ControllerAction::start uses the boolean slot.in_use to see that the thread is still running. see https://github.com/magazino/move_base_flex/blob/master/mbf_abstract_nav/src/controller_action.cpp#L69 But it is used to infer whether we are still in runImpl, i.e., the goal is active and the exe_path controller is processing it.

The race condition is the following sequence:

• runImpl is done
  • Set also the goal to succeeded, for example, because goal is reached
• Another goal comes in, we call ControllerAction::start for it
• ControllerAction::start sees slot.in_use==true
• ControllerAction::start wants to cancel the goal that was succeeded, yields
  • [/navigation/move_base_flex ERROR 1646052687.690027]: To transition to a cancelled state, the goal must be in a pending, recalling, active, or preempting state, it is currently in state: 3
  • We update the internal structure for the thread about to terminate, and critical, also we accept the new goal but no one is going to process the goal
  • If I understand the logic in AbstractActionBase::start this goal is then also never cancelled upon a potential new goal arriving.
• Run sets now slot.in_use=false

Unfortunately, the node was not running with a log-level of debug when we hit the race condition. So I do not have the log about "Finished action ...".