search

usdot-fhwa-stol / carma-platform

CARMA Platform is built on robot operating system (ROS) and utilizes open source software (OSS) that enables Cooperative Driving Automation (CDA) features to allow Automated Driving Systems to interact and cooperate with infrastructure and other vehicles through communication. Doxygen Source Code Documentation :
https://usdot-fhwa-stol.github.io/documentation/carma-platform/
403 stars 124 forks source link

BSM generator is failing to generate valid BSM #2402

Open MishkaMN opened 5 months ago

MishkaMN commented 5 months ago

Summary

For some runs BSM was not able to be encoded. The issue is intermittent, but sometimes happens a lot to make BSM unusable. I have seen the issue in simulation where the vehicle did not move at all. It suddenly started not being able to encode BSM, which the log shows failure to create projection from x,y.z to lat and lon that's needed in BSM. See below picture:

image

And also I noticed this issue in real life vehicle where BSM's lat lon comes out wrong intermittently:

good: image few sec later where vehicle didn't move, bsm id not changed (because configured to be fixed): image

This is most likely occurring from constantly recreating projector because the georeference msg is being published at some rate and there is a race condition occurring:

  void BSMGenerator::georeferenceCallback(const std_msgs::msg::String::UniquePtr msg)
  {
    // Build projector from proj string
    map_projector_ = std::make_shared<lanelet::projection::LocalFrameProjector>(msg->data.c_str());
  }

NOTE: It probably didn't have issue in ROS1 because we had 0.5hz as publication before. But ROS2 migration made it 2hz (every 500ms), which also probably should be reverted back: ROS1: https://github.com/usdot-fhwa-stol/autoware.ai/blob/carma-system-4.1.0/common/map_file/nodes/map_param_loader/map_param_loader.cpp#L98 ROS2: https://github.com/usdot-fhwa-stol/autoware.ai/blob/carma-system-4.5.0/common/map_file_ros2/nodes/map_param_loader/map_param_loader.cpp#L63

Version

4.5.0 (Current)

Expected Behavior

BSM is able to be encoded

Actual Behavior

BSM not encodeable

Steps to Reproduce the Actual Behavior

Start carma and wait for logs to start saying not able to encode BSM. Issue is intermittent. Perhaps it will occur more on PC that has less power.

Related Work

No response

MishkaMN commented 5 months ago

Turning the georeference broadcast back to 2Hz actually makes the startup process little unstable where platform shuts down soon after startup. This is because when some data is broadcasting such as MAP, the node may not have georeference when it is instructed to process it. Then the common handling method looks like just throw an exception. Instead, the callers probably need to handle the exception gracefully