LTP/UDP rate limiting

[BrianSipos]

Neither LTP nor UDP provide intrinsic rate limiting behavior. When operating LTP-over-UDP over a known-rate link it is helpful for the LTP engine itself to rate limit segments to individual peer engines. The limit could be as simple as a Token Bucket for each peer that decrements by the segment size when one is sent and increments at a desired rate (the configure parameter) and refuses to send a segment until its size is available. This is really to limit data segments, so the other segment types could be counted in decrementing but ignored for send gating (to avoid unnecessary report or ACK delays).

I'm planning on implementing a simple rate limit for prototyping, so this ticket is just to inform that this work is happening outside of the normal HDTN project. I'd be happy to coordinate changes later on if it is working properly.

[briantomko]

Rate limiting was attempted on UDP only in https://github.com/nasa/HDTN/blob/master/common/udp/src/UdpBundleSource.cpp using https://github.com/nasa/HDTN/blob/master/common/util/src/RateManagerAsync.cpp It only worked well for very low rates. High rates did not work well due to the resolution/precision of when timers would expire. I haven't considered the Token Bucket, but the modified variation of it may work using the r*S/1000.

[BrianSipos]

What kinds of order of magnitude do you mean by low or high rate?

I implemented a working rate limiter by splitting the "rate" value into a separate numerator (byte count) from denominator (time duration) and using boost duration arithmetic and the actual inter-packet timing rather than using a work ioservice/thread. As long as the system clock has resolution enough to actually capture inter-packet delay this seems to work well. For high rates the denominator can be lowered to avoid rounding artifacts (down to an overall fill rate of bytes-per-microsecond [or nanosecond if boost timer supports it]). I could share a cleaned up diff on email if that would be helpful.

[briantomko]

Yes if you could please share that with me that would be helpful.

[BrianSipos]

Since there have been many (really good and quick) changes in the main branch since my last edits, I'll send you a new diff shortly containing the TokenRateLimiter addition that I am using.

[briantomko]

@BrianSipos This issue has been resoved with the merge of branch 78-udp-token-bucket-rate-limiter into master.

API change: A (last) parameter has been added to bool AddLtpUdpEngine(..., const uint64_t maxSendRateBitsPerSecOrZeroToDisable)

A variation of your TokenRateLimiterwas added called BorrowableTokenRateLimiter. It uses signed numbers and can go negative (if there is at least 1 token available to the borrower). I was able to obtain more accurate rates with the signed version. The tokenMaxLimitDurationWindow is 100 milliseconds (as in your Test examples) (third parameter of SetRate). However, a timer will add time to the limiter every 20 milliseconds. These values are static constants at the top of LtpEngine.cpp.

Just FYI we are also using your TokenRateLimiterin our UDP convergence layer as well. Thanks for your contribution.

[BrianSipos]

Oh great. When I integrated it the non-data segments counted against the token count but weren't gated for the count. So that setup could have gone below zero and I didn't specifically test that situation. i.e. the count should be a signed value to avoid subtraction overflow.