bumping up "limit" at higher bandwidths

[dtaht]

For some traffic, at higher speeds than the original scenarios sqm-scripts was targetted for, the default limit can be too low. As to what it should be set to, I dunno, but I can get it to crack 1000 packets outstanding with the right forms of traffic at >200mbps. I have seen 2000 at a gbit, or more.

having a larger limit generally does not hurt if you have sufficient memory.

This is also in relation to the recent bug discussed and fixed in the thread:

https://www.mail-archive.com/netdev@vger.kernel.org/msg108623.html

I do not think we need to expose this new parameter. Do need to backport it to openwrt tho.

[moeller0]

Hi Dave,

On May 3, 2016, at 03:48 , Dave Täht notifications@github.com wrote:

For some traffic, at higher speeds than the original scenarios sqm-scripts was targetted for, the default limit can be too low. As to what it should be set to, I dunno, but I can get it to crack 1000 packets outstanding with the right forms of traffic at >200mbps. I have seen 2000 at a gbit, or more.

having a larger limit generally does not hurt if you have sufficient memory.

Sure, we could default to scaling this by the amount of available memory and the link-speed/shaped speed on the interface and the number of fq_codel instances on the device.

grep -e MemTotal /proc/meminfo will give the size of the ram on openwrt as well as (at least some flavors of) linux, sqm knows the maximum of ingress and egress shaper bandwidth. So we could simply say if MemTotal < 128MB limit = 1001, otherwise limit = 10240 (the real fq_codel default) Worstcase memory consumption per fq_codel instance for 10240 packets is (10240 * 2048) / (1024^2) = 20 MB, so with 64 MB ram this might lead to OOM/reboots, with 128 MB I assume 20 MB is not too expensive unless we have many fq_codels instantiated. At 1001 packet we only face around 2 MB which should even work for a 32MB device. (I realize that Dave taught cerowrt to use smaller than 2K skbs for smaller packets, but I assume at 1514Byte packets we still require 2048 byte “true-size” per packet) The other dimension to consider is bandwidth, at 100 Mbps we might get away with 1000 packets, so maybe we just linearly scale limit between 100 Mbps and 1000Mbps (10Mbps?) between 1000 and 10240?

Maybe we should turn this around and simply configure a fixed amount of worst case memory consumption budget and split this over the available fq_codel instances?

This is also in relation to the recent bug discussed and fixed in the thread:

https://www.mail-archive.com/netdev@vger.kernel.org/msg108623.html

I do not think we need to expose this new parameter. Do need to backport it to openwrt tho.

I believe Eric sets the default to 64 packets (sort of using a shovel instead of a spoon to get the “manure” out more quickly under saturating load) so all we need to do is wait for the kernel backport… and we will get the 64 by default.

Best Regards

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub

[dtaht]

you missed the GRO problem in your calculations. I have seen 64K sized packets on the armada 385, for example, and while we fixed openwrt to reduce the size of the slab for small packets, that never made it upstream. Anyway, for rates between 200Mbit and a gbit more testing is needed, and I do tend to agree that outer limit should be scaled by the available memory on the system. 10240 is WAY too much on a GRO system.

(cake does this WAY more right).

[moeller0]

Hi Dave,

On May 5, 2016, at 06:13 , Dave Täht notifications@github.com wrote:

you missed the GRO problem in your calculations. I have seen 64K sized packets on the armada 385, for example, and while we fixed openwrt to reduce the size of the slab for small packets, that never made it upstream. Anyway, for rates between 200Mbit and a gbit more testing is needed, and I do tend to agree that outer limit should be scaled by the available memory on the system. 10240 is WAY too much on a GRO system.

I guess each GRO packet should be accounted for as the number of packets they were merged from; I believe there is something like skb->fragments or so that contains that information…

(cake does this WAY more right).

I agree; Toke had a valid point though, that for OOM protection specifying the actual required buffer size is ideal, while for people want to adjust buffers to allow for large bandwidth delay products really want to be able to specify the number of packets (assuming no GRO…). With decent AQM I believe just specifying the memory limit as the true-size should be sufficient though (especially as currently truesize should be equal to 2048Byte, so memory limit/2K should be equal to the number of de-aggregated packets pretty much the number required for the BDP sizing, largely independent of GRO…

Best Regards

— You are receiving this because you commented. Reply to this email directly or view it on GitHub