dora-rs / dora

DORA (Dataflow-Oriented Robotic Application) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.
https://dora-rs.ai
Apache License 2.0
1.35k stars 68 forks source link

Error when a node receives two data streams #502

Open RuPingCen opened 1 month ago

RuPingCen commented 1 month ago

We try to use node C to receive data streams sent by two different nodes A and B. Sometimes the receiving node C will have the error shown in the figure below,

[c node] received unexpected event: 2

1

This results in the functions on lines 126 and 130 of the code never being called, as shown in the figure below

2

Here is the dataflow.yaml

- id: controltest
    custom:
      source: controldata
      inputs:
        tick: dora/timer/millis/100
      outputs:
        - road_lane

  - id: road_lane
    custom:
      source: ../../map/road_line_publisher/build/road_lane_publisher_node
      inputs:
              DoraGnssPose: gnss_poser_sub/DoraGnssPose
              road_lane: controltest/road_lane
      outputs:
            - cur_pose_all

  - id: planning
    custom:
      source: ../../planning/routing_planning/build/routing_planning_node
      inputs:
              cur_pose_all: road_lane/cur_pose_all
              road_lane: controltest/road_lane
      outputs:
            - raw_path
phil-opp commented 1 month ago

Some nodes seem to exit with a -1 return value. What is the cause for that?

[c node] received unexpected event: 2

Event type 2 is an InputClosed event. Dora sends this event when one of a node's inputs is closed. Inputs are closed when the source node exits.

RuPingCen commented 1 month ago

Thanks for your reply. I wrote a c_node test example.

https://github.com/dora-rs/autoware.universe/commit/e10fd19a7140492f36bc05a395eebec18c006338

Here, nodes A and B publish data at 10Hz and 100Hz respectively. Node C receives data from nodes A and B, and no such error occurs. This does not seem to be a dora kernel error.

I will continue to check my code according to your suggestion.

phil-opp commented 1 month ago

Could you share the full output from your out file as well please? I tried to run your example code but I don't see any errors apart from some "stream did not contain valid UTF-8" errors.

phil-opp commented 1 month ago

I opened https://github.com/dora-rs/dora/pull/510 to make non-UTF8 output from nodes non-fatal. This way, stdout stays open and no SIGPIPE occurs. During my testing, this SIGPIPE was the reason that node_C failed early.

The reason for the non-UTF8 output of node_C is the that the string returned by read_dora_input_id is not zero-terminated. So you have to convert it to a C++ std::string first using std::string id(data_id, data_id_len);. Otherwise we will continue reading the random bytes behind the string until the next null byte. So to fix your node C:

- std::cout << "Input Data length: " << data_len << "   " << data_id << std::endl;
+ std::string id(data_id, data_id_len);
+ std::cout << "Input Data length: " << data_len << "   " << id << std::endl; 

Please let me know whether this fixes your issue! If there are still other issues, please share your out folder.