Maximum size of action server goal (guestimate)?

[gavanderhoorn]

(this is more a Q&A than an actual issue, but Discussions are not enabled, so my apologies for posting it as an issue)

Would you happen to know, off the top of your head, what the maximum size would be for an action goal to still make it to a Client, via an Agent? I'm specifically referring to an action goal here, but as that's transmitted via services, that'd also be OK I believe.

This is related to #217 and #247, as while we've been able to get everything to work with your advice, we're trying to determine the limits to various things, and notice action goals for which the Micro-ROS Agent prints [==>> DDS <<==] [...] len: 67776 [...] don't make it to the Client (a FollowJointTrajectory action goal in this specific case).

There is no warning or error printed AFAICT, the messages just seem to vanish. No traffic to the Client either (Wireshark).

Before diving into the various packages, I thought I'd ask, as you may already be able to point us in a certain direction.

Context: UDPv4 Client, UCLIENT_TWEAK_XRCE_WRITE_LIMIT=OFF, UCLIENT_UDP_TRANSPORT_MTU=2048, RMW_UXRCE_MAX_HISTORY=4 and RMW_UXRCE_STREAM_HISTORY=32.

thanks

---

Edit: the Client application should have been given ample space to receive the goal (by configuring a micro_ros_utilities_memory_conf_t with values for size higher than the goal being sent uses/needs).

[pablogs9]

Some details about DDS-XRCE.

DATA sub-messages are exchanged between the client and the agent in order to transmit DDS payloads from the client to the agent (in publishers, requesters, and repliers) or from the agent to the client (when dealing with subscriptions, requesters, and repliers).

This DDS-XRCE DATA sub-message has a lenght field in its subheader of 16 bits, that means that the maximum payload will be about 65 kB (with an MTU that allows that).

When dealing with fragmentation, the same happens because the whole DDS-XRCE DATA sub-message (subheader + payload) will be chopped into smaller fragments. That means that the maximum payload is still about 65 kB even if the fragmentation is used.

What happens when a micro-ROS client tries to publish a big message? It depends:

1. If size < MTU it fits in an XRCE message and will be sent as it
2. If size > MTU && size < (MTU*UXRCE_OUTPUT_STREAM_HISTORY), the message will be chopped into fragments and each fragment will be allocated in a different XRCE message and the whole buffer (all XRCE messages) will be flushed to the agent and the agent will reconstruct the original DATA submessage from the FRAG submessages.
3. If size > MTU && size > (MTU*UXRCE_OUTPUT_STREAM_HISTORY) && size < 2^16 B the message will be chopped into fragments but the fragments do not fit in the buffer so, fragments will be prepared, the serializer will start serializing data and every time the XRCE output buffer is full the FRAG messages will be flushed to the agent. The agent will know when it should stop saving fragments and should regenerate the DATA message from FRAGs because the final frag has a bit enabled in its header. This is known as continuous serialization mode, and you can see the implementation here and here. Agent will know that the regenerated DATA submessage will be ok if all the FRAG payloads matches the DATA length in the DATA subheader.
4. If size > MTU && size > (MTU*UXRCE_OUTPUT_STREAM_HISTORY) && size > 2^16 B Same as above but the DATA length is set to 0, so the agent needs to rely on receiving all fragments correctly (and this should be like this because XRCE reliability) and will assume that the regenerated DATA submessage is correct. The final FRAG will have the last_fragment_bit enabled. This is the UCLIENT_TWEAK_XRCE_WRITE_LIMIT, and it is off-standart.

What happens when a micro-ROS Agent tries to send a DATA payload to micro-ROS client:

1. If size < MTU it fits in an XRCE message and will be sent as it.
2. If size > MTU && size < (MTU*UXRCE_INPUT_STREAM_HISTORY), the message will be chopped into fragments and each fragment will be allocated in a different XRCE message and the whole buffer (all XRCE messages) will be flushed to the client and the client will reconstruct the original DATA submessage from the FRAG submessages.
3. If size > MTU && size > (MTU*UXRCE_INPUT_STREAM_HISTORY), same as above but the client will fill its buffers with the first UXRCE_INPUT_STREAM_HISTORY FRAGs and the FRAG number UXRCE_INPUT_STREAM_HISTORY+1 will be never acknowledged because it does not fit in the client's memory. So the agent will be trying to retrieve the client's acknowledge of the FRAG UXRCE_INPUT_STREAM_HISTORY+1 but it will never be received.

Headers are not taken into account in the above formulas, so they are not literal. Just a reference.

As we can see there is a couple of problems here:

1. The client does not have dynamic memory so it won't be able to accept DATA messages split in more FRAG than UXRCE_INPUT_STREAM_HISTORY
2. If the agent tries to send more FRAGs than UXRCE_INPUT_STREAM_HISTORY the client buffer will be blocked. @Acuadros95 we should take a look on that.

Answering your question @gavanderhoorn, the agent won't send a DATA payload (fragmented or not) to the client if it greater than 2^16 B because of standard limitation.

[gavanderhoorn]

Thanks for the extensive info @pablogs9.

I'd understood max payload is 64K, also based on our use of UDPv4 in the client.

But the intricacies of how MTU and the other variables interact were beyond my current understanding.

Thanks again

2. If the agent tries to send more FRAGs than UXRCE_INPUT_STREAM_HISTORY the client buffer will be blocked. @Acuadros95 we should take a look on that.

will you open an issue for this, or would you want me to open one?

1. The client does not have dynamic memory so it won't be able to accept DATA messages split in more FRAG than UXRCE_INPUT_STREAM_HISTORY

This may be something we could change (ie: enable dynamic memory). But for something like a FollowJointTrajectory goal, it will essentially mean we can accept goals up-to a size which would fit in memory.

Would something like the continuous publication, but then for subscription be possible? Deserialisation would perhaps be difficult to implement, and not all messages would be compatible (dependencies between fields), but it could help overcome these problems.

[pablogs9]

will you open an issue for this, or would you want me to open one?

Feel free to open an issue.

Would something like the continuous publication, but then for subscription be possible? Deserialisation would perhaps be difficult to implement, and not all messages would be compatible (dependencies between fields), but it could help overcome these problems.

This solution may be possible, but currently, there are no plans for implementing any solution in that sense.

[gavanderhoorn]

will you open an issue for this, or would you want me to open one?

Feel free to open an issue.

which tracker should I open this issue on? The Micro-ROS-Agent repository?

I believe you have a better insight into what the actual issue is.

I'd be willing to open one, but my summary of the issue would probably be "messages larger than a certain size seem to 'disappear'".

[pablogs9]

I'd be willing to open one, but my summary of the issue would probably be "messages larger than a certain size seem to 'disappear'".

That's enough

micro-ROS Agent is a good place. Mention this issue as related.

Thanks!

[gavanderhoorn]

I've opened https://github.com/micro-ROS/micro-ROS-Agent/issues/143.

[pablogs9]

Can we close this in favor of the new one @gavanderhoorn ?

[gavanderhoorn]

Yep.

thanks for the guidance.