Perception related components not running correctly

[ThankPan]

Describe the bug

We have been running Apollo with Carla aiming to test the complete task pipeline. Therefore, we noticed that the perception module is not actually running. The objects in the "/apollo/perception/obstacles" are actually generated by the ObjectSensor which writes the info and bounding boxes to the channel. With these pseudo objects, the prediction started running.

We then commented out the ObjectSensor since we want the perception modules such as DetectionComponent or the CameraObstableDetectionComponent to generate obstacle info based on sensor output for prediction. But we found that after doing this the vehicle wouldn't move with nothing in the PerceptionObstacle as input for prediction. After examining the dag file started by perception, we also found that the read and write channels of RecognitionComponet, Fusion, and V2XFusion are mostly empty(Certain inner channels are not visible in cyber_channel).

We wonder what is the cause of this problem. Some modification on Apollo? or the ObjectSensor is crucial? The vehicle should be able to drive even without obstacles in the simulation environment.

The release we used was 0.2.0 due to our Apollo version is 7.0. Could the 0.3.0 release still support 7.0 and solve this problem?

What version of carla_apollo_bridge?

0.2.x

[XiaoFei9704]

We are fixing the problem with the perception module, and it will definitely be fixed within this month. If there is no object_sensor, the car should be able to run. Logically speaking object_sensor is not necessary. I will take a look at this issue. Finally, version 0.3.0 of the bridge can only support the simulation of Apollo 8.0 and Carla.

[ThankPan]

We are fixing the problem with the perception module, and it will definitely be fixed within this month. If there is no object_sensor, the car should be able to run. Logically speaking object_sensor is not necessary. I will take a look at this issue. Finally, version 0.3.0 of the bridge can only support the simulation of Apollo 8.0 and Carla.

Do you mind sharing some thoughts or solutions on fixing the perception modules? We would be happy to contribute since it could also help our project process which relies on this feature. We found that the channel sensor/velodyne128 which the DetectionComponent reads from is functioning as normal, but no output is visible since it writes to an inner channel. That's where our debugging stopped and couldn't locate the problem source.

[ThankPan]

The problem of vehicle not moving when ObjectSensor was disabled has been solved after looking closely into the logs, finding that the required Transform info was missing due to not starting the module. For those who want to use the original perception of Apollo instead of pseudo objects, the modules required to start(at least these modules) is listed as below: Perception, Prediction, Planning, Control, Localization, Routing and Transform.

However, I haven't tested with real obstacles on the road yet and there are still small bugs existing, such as some bounding boxes flashing. Therefore, I might not close this issue yet.

Another crucial problem is that certain lane info is missing for Apollo and is causing the vehicles stuck at roadside when turning or after turning. I wonder is there any possible solution fixing this?

[Michael-Fuu]

I found a PerceptionObstacle problem: when using Apollo Percetion module instead of carla real obstacle information, alrough the detection results are poor by apollo's default percetion model, the location of detection results is wired, always at a opposite location. So I wonder maybe there are bugs in bridge. As shown on the graph, the person is on the right while the detection result showing obstacle is on the left.

[ThankPan]

I found a PerceptionObstacle problem: when using Apollo Percetion module instead of carla real obstacle information, alrough the detection results are poor by apollo's default percetion model, the location of detection results is wired, always at a opposite location. So I wonder maybe there are bugs in bridge. As shown on the graph, the person is on the right while the detection result showing obstacle is on the left.

You can't use the modified code in #104. The modification in https://github.com/guardstrikelab/carla_apollo_bridge/issues/104#issuecomment-1569570789 would cause the point cloud coordinates error. Although the point cloud demonstration in Dreamview is normal, but is just useless to perception. So I changed back to the commented out part and the perception bounding boxes are in normal position.

[Michael-Fuu]

The problem of vehicle not moving when ObjectSensor was disabled has been solved after looking closely into the logs, finding that the required Transform info was missing due to not starting the module. For those who want to use the original perception of Apollo instead of pseudo objects, the modules required to start(at least these modules) is listed as below: Perception, Prediction, Planning, Control, Localization, Routing and Transform.

However, I haven't tested with real obstacles on the road yet and there are still small bugs existing, such as some bounding boxes flashing. Therefore, I might not close this issue yet.

Another crucial problem is that certain lane info is missing for Apollo and is causing the vehicles stuck at roadside when turning or after turning. I wonder is there any possible solution fixing this?

Hello, can you help me? my localization satus is WARN, the warning logs is :gps/imu frame Delay!Cannot find Matching IMU. IMU messages too old. Newest timestamp[1.70427e+09], GPS timestamp[1.70427e+09]. So I wonder which algorithm you choose for localization? As I can't launch msf so I choose rtk and get warnning.

[ThankPan]

The problem of vehicle not moving when ObjectSensor was disabled has been solved after looking closely into the logs, finding that the required Transform info was missing due to not starting the module. For those who want to use the original perception of Apollo instead of pseudo objects, the modules required to start(at least these modules) is listed as below: Perception, Prediction, Planning, Control, Localization, Routing and Transform. However, I haven't tested with real obstacles on the road yet and there are still small bugs existing, such as some bounding boxes flashing. Therefore, I might not close this issue yet. Another crucial problem is that certain lane info is missing for Apollo and is causing the vehicles stuck at roadside when turning or after turning. I wonder is there any possible solution fixing this?

Hello, can you help me? my localization satus is WARN, the warning logs is :gps/imu frame Delay!Cannot find Matching IMU. IMU messages too old. Newest timestamp[1.70427e+09], GPS timestamp[1.70427e+09]. So I wonder which algorithm you choose for localization? As I can't launch msf so I choose rtk and get warnning.

I also encountered this warning and ignored it. I'm not clear about the effect it has on the performance but it didn't affect my driving.

[wqklihbf]

在仔细查看日志后，解决了禁用 ObjectSensor 时车辆不移动的问题，发现由于未启动模块而丢失了所需的 Transform 信息。对于那些想使用阿波罗的原始感知而不是伪对象的人来说，启动所需的模块（至少这些模块）如下：感知、预测、规划、控制、定位、路由和转换。 但是，我还没有测试过道路上的真正障碍物，仍然存在一些小错误，例如一些边界框闪烁。因此，我可能还没有关闭这个问题。 另一个关键问题是，阿波罗缺少某些车道信息，导致车辆在转弯时或转弯后卡在路边。我想知道是否有任何可能的解决方案来解决这个问题？

您好，你能帮我吗？我的本地化 satus 是 WARN，警告日志是：gps/imu frame Delay！找不到匹配的 IMU。IMU 消息太旧。最新时间戳[1.70427e+09]，GPS时间戳[1.70427e+09]。所以我想知道你选择哪种算法进行本地化？由于我无法启动 msf，所以我选择了 rtk 并收到警告。

Hello, I have the same problem. Have you solve the problem?

[YangXianqi]

我发现了一个 PerceptionObstacle 问题：当使用 Apollo Percetion 模块而不是 carla 真实的障碍物信息时，虽然 apollo 默认的感知模型检测结果很差，但检测结果的位置是连线的，总是在相反的位置。所以我怀疑 bridge 可能存在错误。 如图所示，人是在右边，而显示障碍物的检测结果在左边。

您好，我想询问一下，如何启用apollo的perception模块，目前我启动后会报时间戳不一致的问题，具体可见https://github.com/guardstrikelab/carla_apollo_bridge/issues/207