Open TanmayDeshmukh opened 2 years ago
Looks like there is some incorrect behavior either
Running the agent in debug mode shows that the request of the AddTwoInts
, two int64_t
integers of 8 bytes each, is transmitted in a UDP packet containing 68 bytes (8 for the header + 60 for the data), thus 60 bytes of data instead of 16. The values of variables int64_t a
and int64_t b
are present in the last 16 bytes of the UDP packet. I don't know if this is intended behavior, but it is not matching the deserialization happening in the MCU side.
The MCU receives the UDP packet when the service is called and writes the last 32 bytes in a buffer. This can be validated by adding a breakpoint at rmw_microxrcedds/rmw_microxrcedds_c/src/rmw_request.c#L124 and validating the contents of static_buffer->buffer
and static_buffer->length
.
The buffer is then sent for deserialization, but it deserialized incorrectly since it expects the data to be at the beginning of the buffer and, in this case, the data is present after a 16 bytes offset.
I still need to dig a little deeper, but there might be a similar issue (namely inconsistent serialization/deserialization) happening on the response. I say this because running the agent in debug mode clearly shows that the agent receives a UDP packet containing ~the~ a response. The data, int64_t sum
, is present in the last 8 bytes of a 32 bytes UDP packet (8 bytes header + 16 bytes unknown + 8 bytes data).
I am pasting below a sample of the exchange reported by the agent when a service is called. Note that I have manually shifted the buffer in rmw_microxrcedds/rmw_microxrcedds_c/src/rmw_request.c#L122, i.e. changed the line to static_buffer->buffer + 16
to read from the correct address.
[1647331268.554430] debug | Replier.cpp | read_fn | [==>> DDS <<==] | client_key: 0x00000000, len: 56, data:
0000: 35 A7 10 01 E7 EC 5B 14 9F 07 04 2B 00 00 14 03 00 00 00 00 01 00 00 00 A3 3D 25 01 73 6C D9 63
0020: 01 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00
[1647331268.554594] debug | UDPv4AgentLinux.cpp | send_message | [** <<UDP>> **] | client_key: 0x3C4C11D9, len: 68, data:
0000: 81 80 02 00 09 01 3C 00 00 0C 00 08 35 A7 10 01 E7 EC 5B 14 9F 07 04 2B 00 00 14 03 00 00 00 00
0020: 01 00 00 00 A3 3D 25 01 73 6C D9 63 01 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00 07 00 00 00
0040: 00 00 00 00
[1647331268.555816] debug | UDPv4AgentLinux.cpp | recv_message | [==>> UDP <<==] | client_key: 0x3C4C11D9, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 03 00 00 00 80
[1647331272.343187] debug | UDPv4AgentLinux.cpp | recv_message | [==>> UDP <<==] | client_key: 0x3C4C11D9, len: 44, data:
0000: 81 80 03 00 07 01 24 00 00 0D 00 08 35 A7 10 01 E7 EC 5B 14 9F 07 04 2B 00 00 14 03 00 00 00 00
0020: 01 00 00 00 A4 3D 25 01 73 6C D9 63
[1647331272.343431] debug | Replier.cpp | write | [** <<DDS>> **] | client_key: 0x00000000, len: 32, data:
0000: 35 A7 10 01 E7 EC 5B 14 9F 07 04 2B 00 00 14 03 00 00 00 00 01 00 00 00 A4 3D 25 01 73 6C D9 63
[1647331272.343553] debug | UDPv4AgentLinux.cpp | send_message | [** <<UDP>> **] | client_key: 0x3C4C11D9, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 04 00 00 00 80
I just noticed that the IP header is also included (minimum 20 bytes and maximum 60 bytes depending on the options). So, could it be that those "extra" bytes are part of the header being misinterpreted as data?
Following up on this, the latest news is that the services work as expected if we change the middleware implementation by prefixing the commands with RMW_IMPLEMENTATION=rmw_fastrtps_cpp
, e.g., RMW_IMPLEMENTATION=rmw_fastrtps_cpp ros2 run micro_ros_agent micro_ros_agent [options]
. Naturally, the same could be achieved by changing the environment variable.
If we do not change to FastRTPS, then the default one in Galactic is used, namely Cyclone DDS. I am showing below the difference in the exchange, as reported by the agent. As it turns out, it looks like the actual issue here is that the DDS message is formatted differently when using Cyclone DDS and this is not accounted for / understood by the agent. My knowledge of DDS is limited, so I do not really know what the additional 16 bytes present in the DDS message from Cyclone DDS stand for. But it really looks like this is the root cause of both issues.
## FastDDS
## Sending service request
[1647417016.074674] debug | Replier.cpp | read_fn | [==>> DDS <<==] | client_key: 0x00000000, len: 40, data:
0000: 01 0F 6F 52 0E 2D 30 C4 01 00 00 00 00 00 12 03 00 00 00 00 01 00 00 00 08 00 00 00 00 00 00 00
0020: 07 00 00 00 00 00 00 00
[1647417016.074733] debug | UDPv4AgentLinux.cpp | send_message | [** <<UDP>> **] | client_key: 0x1632FAA9, len: 52, data:
0000: 81 80 03 00 09 01 2C 00 00 0C 00 08 01 0F 6F 52 0E 2D 30 C4 01 00 00 00 00 00 12 03 00 00 00 00
0020: 01 00 00 00 08 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00
[1647417016.075906] debug | UDPv4AgentLinux.cpp | recv_message | [==>> UDP <<==] | client_key: 0x1632FAA9, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 04 00 00 00 80
## Receiving service response
[1647417058.588360] debug | UDPv4AgentLinux.cpp | recv_message | [==>> UDP <<==] | client_key: 0x1632FAA9, len: 44, data:
0000: 81 80 04 00 07 01 24 00 00 0E 00 08 01 0F 6F 52 0E 2D 30 C4 01 00 00 00 00 00 12 03 00 00 00 00
0020: 01 00 00 00 0F 00 00 00 00 00 00 00
[1647417058.588563] debug | Replier.cpp | write | [** <<DDS>> **] | client_key: 0x00000000, len: 32, data:
0000: 01 0F 6F 52 0E 2D 30 C4 01 00 00 00 00 00 12 03 00 00 00 00 01 00 00 00 0F 00 00 00 00 00 00 00
[1647417058.588614] debug | UDPv4AgentLinux.cpp | send_message | [** <<UDP>> **] | client_key: 0x1632FAA9, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 05 00 00 00 80
### Cyclone
## Sending service request
[1647417175.149524] debug | Replier.cpp | read_fn | [==>> DDS <<==] | client_key: 0x00000000, len: 56, data:
0000: 21 F8 10 01 2B 13 F2 1B 13 0C 69 22 00 00 14 03 00 00 00 00 01 00 00 00 70 EF FA 99 A6 24 60 85
0020: 01 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00
[1647417175.149686] debug | UDPv4AgentLinux.cpp | send_message | [** <<UDP>> **] | client_key: 0x28E48004, len: 68, data:
0000: 81 80 02 00 09 01 3C 00 00 0C 00 08 21 F8 10 01 2B 13 F2 1B 13 0C 69 22 00 00 14 03 00 00 00 00
0020: 01 00 00 00 70 EF FA 99 A6 24 60 85 01 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 07 00 00 00
0040: 00 00 00 00
[1647417175.150904] debug | UDPv4AgentLinux.cpp | recv_message | [==>> UDP <<==] | client_key: 0x28E48004, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 03 00 00 00 80
## Receiving service response
[1647417204.691133] debug | UDPv4AgentLinux.cpp | recv_message | [==>> UDP <<==] | client_key: 0x28E48004, len: 44, data:
0000: 81 80 03 00 07 01 24 00 00 0D 00 08 21 F8 10 01 2B 13 F2 1B 13 0C 69 22 00 00 14 03 00 00 00 00
0020: 01 00 00 00 71 EF FA 99 A6 24 60 85
[1647417204.691417] debug | Replier.cpp | write | [** <<DDS>> **] | client_key: 0x00000000, len: 32, data:
0000: 21 F8 10 01 2B 13 F2 1B 13 0C 69 22 00 00 14 03 00 00 00 00 01 00 00 00 71 EF FA 99 A6 24 60 85
[1647417204.691584] debug | UDPv4AgentLinux.cpp | send_message | [** <<UDP>> **] | client_key: 0x28E48004, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 04 00 00 00 80
Issue template
Steps to reproduce the issue
ros2 run micro_ros_setup configure_firmware.sh add_two_ints_service --transport udp --ip <our agent ip> --port 8888
ros2 service call /addtwoints example_interfaces/srv/AddTwoInts "{a: 5, b: 10}"
Expected behavior
Should get a response (sum=15)
Actual behavior
waiting for service to become available... requester: making request: example_interfaces.srv.AddTwoInts_Request(a=5, b=10)
Additional information
We basically have the same issue as this but with FreeRTOS on our STM32 board. We using a router from Teltonika- RUT240 (which a decent router), and the computer running the agent and making service calls is connected though Ethernet.
Quick note: I had to include
#include <rclc/executor.h>
in the example code, and change "printk"s to printf to get it to compile.Not very well-versed with uROS or DDS stuff, but we tried the following:
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
before starting the agent, as suggested in this thread.Logs from the agent: