Transfer Rate

[fattaholmanan]

Hi, Recently I got to know norm and I'm giving it a try. Apparently it has a TCP-friendly congestion control. However, when I'm trying to use the normCast example, the sending rate doesn't go beyond 11.2 kbps. Is it normal? I'm running the commands in mininet. Here are the commands I'm using:

sender:

./normCast id 7 send u.pdf addr 172.16.0.100 cc

receiver:

./normCast id 10 recv ./

part of the debugging info in the sender:

Proto Info: ProtoDebug>SetDebugLevel: debug level changed from 1 to 10
Proto Debug: NormSession::SetTxRateInternal() node>7 increased to new grtt to: 1.064000 sec
Proto Debug: SenderRateTracking time>1638473409.821098 rate>11.200000 rtt>0.001047 loss>0.000000
normCast: enqueued "u.pdf" for transmission ...
Proto Debug: SenderRateTracking time>1638473410.885716 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473411.950401 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473413.013693 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473414.078423 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473415.141370 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473416.205700 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473417.270224 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473418.333997 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638473419.398407 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Info: REPORT time>19:30:19.774828 node>7 ***************************************
Proto Info: Local status:
Proto Info:    txRate>   11.200 kbps sentRate>   11.684 grtt>1.064000
Proto Info: ***************************************************************************

[bebopagogo]

I think the issue is the feedback from your receiver is not getting to your sender. So, the rate is not increasing since there is no feedback.

The default session address that the normCast example uses is a multicast address (224.1.2.3) and the receiver sending its acknowledgments to that multicast address but your sender is configured for a unicast session address.

A work-around for this is to change your receiver command-line to:

./normCast id 10 addr 127.0.0.1 recv ./

This puts the receiver into a unicast mode so that it sends its feedback to the detected unicast address for the sender. The "127.0.0.1" loopback address then a sort of "dummy address" as the receiver will receive NORM packets from any sender and unicast feedback to each sender's (if there were multiple senders pointing at the receiver) unicast address.

(if you add the "trace" option to your command-lines you can see the packets being sent/received by the normCast instances)

The "normCast" is a simple example ... There is also a "work in progress" NormSocket extension to the NORM API that follows more a typical connection-oriented (TCP like) paradigm and there are "examples/normServer.cpp" and "example/normClient.cpp" that use that API extension. I still have more work to do on that extension and I plan to integrate as a part of the NORM library when I feel it is more complete. It is usable now, but only provides a byte-stream transfer model using the NORM_OBJECT_STREAM ... although you can pipe a file into the STDIN of the "normServer" (or normClient) and the content of that file will be passed out the STDOUT of the associate normClient(s) (or normServer). I plan to further enhance this NormSocket API to also provide NORM_OBJECT_FILE and NORM_OBJECT_DATA as well as the "message stream" option of NORM_OBJECT_STREAM but my starting point was to emulate TCP connection-oriented byte-stream delivery with the NORM API.

cheers,

Brian

[fattaholmanan]

Thank you Brian for the detailed comment. I managed to make the CC work properly using your "work-around" by assigning a unicast IP in the receiver side. However, I'm indeed more curious about the multicast. Is the multicast also follows a TCP friendly CC? How can I send with higher rate using multicast?

[bebopagogo]

Yes,  the multicast operation can also use the TCP friendly CC … You just need to use a multicast session address (e.g., 224.1.2.3)  and often you need to use the “interface” command line option so that the IP multicast group joins and packets are sent on the intended network interface (e.g. “normcast addr 224.1.2.3/6003 interface eth0 recv ./”  and similar on the sender.

As mentioned, the default is for the feedback messages (ACK and NACK) to also be sent via multicast.  That works fine, but sometimes it is more advantageous to have feedback messages from the receivers be sent via unicast back to the sender.  In some of my examples, I include a “unicastFeedback” command-line option to demo that, but have not yet added that to the normCast example.

Best regards,

Brian

From: Ali Fattaholmanan @.> Reply-To: USNavalResearchLaboratory/norm @.> Date: Friday, December 3, 2021 at 8:20 PM To: USNavalResearchLaboratory/norm @.> Cc: Brian Adamson @.>, Comment @.***> Subject: Re: [USNavalResearchLaboratory/norm] Transfer Rate (Issue #48)

Thank you Brian for the detailed comment. I managed to make the CC work properly using your "work-around" by assigning a unicast IP in the receiver side. However, I'm indeed more curious about the multicast. Is the multicast also follows a TCP friendly CC? How can I send with higher rate using multicast?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[fattaholmanan]

For some reasons, I cannot activate the multicast CC the way you described about. I'm using a static multicast group 239.0.0.1 where has h1,h2 and h3 as members.

sender command:

h1$ ./normCast send ../random.file addr 239.0.0.1 debug 10 interface h1-eth0
Proto Info: ProtoDebug>SetDebugLevel: debug level changed from 1 to 10
normCast: auto assigned NormNodeId: 3232235876 (IP address: 192.168.1.100)
Proto Debug: NormSession::SetTxRateInternal() node>3232235876 increased to new grtt to: 1.064000 sec
Proto Debug: SenderRateTracking time>1638584372.141409 rate>11.200000 rtt>0.001047 loss>0.000000
normCast: enqueued "random.file" for transmission ...
Proto Debug: SenderRateTracking time>1638584373.206090 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584374.269705 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584375.333641 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584376.397654 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584377.462701 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584378.525507 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584379.589694 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584380.654095 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Debug: SenderRateTracking time>1638584381.718779 rate>11.200000 rtt>0.001047 loss>0.000000
Proto Info: REPORT time>02:19:42.089825 node>3232235876 ***************************************
Proto Info: Local status:
Proto Info:    txRate>   11.200 kbps sentRate>   11.689 grtt>1.064000
...

receivers commands:

h3$ ./normCast recv ./d3 addr 239.0.0.1 interface h3-eth0
h2$ ./normCast recv ./d2 addr 239.0.0.1 interface h2-eth0

[bebopagogo]

Another thing to check is to make sure there is not a firewall rule in place blocking the NORM/UDP feedback packets back to the sender.

If you use the “trace” option you get debug output describing every packet NORM sends or receives. That’s a way to quickly check that packets are flowing as needed

Sent from my iPhone

On Dec 3, 2021, at 9:33 PM, Ali Fattaholmanan @.***> wrote:

 For some reasons, I cannot activate the multicast CC the way you described about. I'm using a static multicast group 239.0.0.1 where has h1,h2 and h3 as members.

sender command:

h1$ ./normCast send ../random.file addr 239.0.0.1 debug 10 interface h1-eth0 Proto Info: ProtoDebug>SetDebugLevel: debug level changed from 1 to 10 normCast: auto assigned NormNodeId: 3232235876 (IP address: 192.168.1.100) Proto Debug: NormSession::SetTxRateInternal() node>3232235876 increased to new grtt to: 1.064000 sec Proto Debug: SenderRateTracking time>1638584372.141409 rate>11.200000 rtt>0.001047 loss>0.000000 normCast: enqueued "random.file" for transmission ... Proto Debug: SenderRateTracking time>1638584373.206090 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584374.269705 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584375.333641 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584376.397654 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584377.462701 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584378.525507 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584379.589694 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584380.654095 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Debug: SenderRateTracking time>1638584381.718779 rate>11.200000 rtt>0.001047 loss>0.000000 Proto Info: REPORT time>02:19:42.089825 node>3232235876 *** Proto Info: Local status: Proto Info: txRate> 11.200 kbps sentRate> 11.689 grtt>1.064000 ... receivers commands:

h3$ ./normCast recv ./d3 addr 239.0.0.1 interface h3-eth0 h2$ ./normCast recv ./d2 addr 239.0.0.1 interface h2-eth0 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[fattaholmanan]

Yes, you were right about the response NORM multicast packets being dropped. Was there a technical/logical reason to make the feedback data multicast as well? Would it be easy to make it unicast instead?

[bebopagogo]

Multicast feedback is the default in the NORM library code mainly for historical reasons.  There are API calls to set the feedback  to unicast (and I think independently for NACK vs. ACK as desired).  So, for your application, you could make unicast feedback its default.    

Multicast NACK feedback provides for a little lower latency with regards to the timer-based multicast feedback suppression mechanism but at a cost of using more network resources than when unicast feedback is available and configured.  If your use case is relaxed on latency (e.g., big file bulk transfer as the latency from the feedback suppression only really kicks in at the end of object transfer to finalize any missing packets from the final block of data transferred and hence the ARQ timer based on GRTT is a contributor to the time it takes for the final rounds of NACKing and repair transmissions), unicast feedback is probably recommended.  But if you are doing something like messaging and want to minimize latency the multicast feedback may be preferred.  (Note you can further minimize latency at the cost of ARQ efficiency by zeroing out the backoff timers that provide feedback suppression … e.g., if you have a small group size so there’s not a “feedback implosion” problem).  We also actually have some networks where we use NORM where there is no unicast routing.

From: Ali Fattaholmanan @.> Reply-To: USNavalResearchLaboratory/norm @.> Date: Sunday, December 5, 2021 at 12:46 PM To: USNavalResearchLaboratory/norm @.> Cc: Brian Adamson @.>, Comment @.***> Subject: Re: [USNavalResearchLaboratory/norm] Transfer Rate (Issue #48)

Yes, you were right about the response NORM packets being dropped. Was there a technical/logical reason to make the feedback data multicast as well? Would it be easy to make it unicast instead?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[fattaholmanan]

Thank you Brian for the quick responses and detailed comments. I go ahead and close this issue.