vehicle_cmd_gate has flaky test

[xmfcx]

Checklist

• [X] I've read the contribution guidelines.
• [X] I've searched other issues and no duplicate issues were found.
• [X] I'm convinced that this is not my fault but a bug.

Description

• https://github.com/autowarefoundation/autoware.universe/actions/runs/8264643291/job/22608668912?pr=6490#step:8:2488

Expected: (std::abs(lon_jerk)) < (max_lon_jerk_lim * threshold_scale), actual: 1.55397 vs 1.54

This test failed. But non-cuda passed for the same instance.

It passed for many other tests too, this is one instance I've found that has failed.

The related code is here:

https://github.com/autowarefoundation/autoware.universe/blob/59c504b2205a5a6a19c4bdef338557d518b09918/control/vehicle_cmd_gate/test/src/test_filter_in_vehicle_cmd_gate_node.cpp#L261

Expected behavior

Test always passes.

Actual behavior

Test failed this time.

Steps to reproduce

• Sometimes see it fail in build-and-test-differential jobs?
• Maybe run it under heavy load?

Versions

• OS: Ubuntu 22.04
• Github public runner

Possible causes

No response

Additional context

No response

[xmfcx]

@TakaHoribe -san, you seem to be the test author.

The test uses a dt variable, so this is a time dependent test. I didn't analyze it very deeply maybe it fails on very low end hardware very occasionally? I don't know.

cc. @takayuki5168

[xmfcx]

https://github.com/autowarefoundation/autoware.universe/blob/59c504b2205a5a6a19c4bdef338557d518b09918/control/vehicle_cmd_gate/test/src/test_filter_in_vehicle_cmd_gate_node.cpp#L390-L400

Looking into it more, here there are 2 i variables where one shadows the other. This alone shouldn't be the source.

I'm assuming this loop was expected to run every 10ms. But if spin_some takes more time, this can fluctuate.

It might be a good idea to replace it with something like:

for (size_t i = 0; i < 100; ++i) {
  auto start_time = std::chrono::steady_clock::now();

  const bool reset_clock = (i == 0);
  const auto cmd = cmd_generator_.calcSinWaveCommand(reset_clock);
  pub_sub_node_.publishControlCommand(cmd);
  pub_sub_node_.publishDefaultTopicsNoSpin();
  for (int j = 0; j < 20; ++j) {
    rclcpp::spin_some(pub_sub_node_.get_node_base_interface());
    rclcpp::spin_some(vehicle_cmd_gate_node_->get_node_base_interface());
  }

  auto end_time = std::chrono::steady_clock::now();
  std::chrono::milliseconds elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end_time - start_time);

  std::chrono::milliseconds sleep_duration = std::chrono::milliseconds{10} - elapsed;
  if (sleep_duration.count() > 0) {
    std::this_thread::sleep_for(sleep_duration);
  }
}

[xmfcx]

Or we could just increase the threshold_scale from 1.1 to 1.2 😅

https://github.com/autowarefoundation/autoware.universe/blob/59c504b2205a5a6a19c4bdef338557d518b09918/control/vehicle_cmd_gate/test/src/test_filter_in_vehicle_cmd_gate_node.cpp#L258

[xmfcx]

Happened again:

• https://github.com/autowarefoundation/autoware.universe/actions/runs/8279452851/job/22653907151?pr=6490#step:8:12408

[takayuki5168]

@xmfcx Thank you for the analysis and suggestion. I agree with them. I created a PR for the change, so let's see what will happen with this PR. https://github.com/autowarefoundation/autoware.universe/pull/6661

[xmfcx]

Fixed with:

• https://github.com/autowarefoundation/autoware.universe/pull/6665

Things have been green for a while, thanks @takayuki5168 -san!