Create shaper_bufferbloat.rst guide

[SeimusS]

Created a guide for FQ_CoDeL to combat Bufferbloat on OPNsense. There are several posts on forum and internet with wrong settings. This should provide proper configuration with optional tuning of FQ_C.

[SeimusS]

I tried to make the guide bit more informative as well provide additional information on the parameters. As it looks like a lot of users do something without knowing why. Or try to look what a parameter does but its hard to find.

P.S. This is my 1st official pull request ever, if there is something wrong let me know I will try to adjust it to my best knowledge.

[Monviech]

I am using such a setup to fight bufferbloat on my asymmetric DSL. (250/40)

What helped me additionally was creating an additional Shaper rule, that matches "tcp (ACK packets only)" and targeting them with an own "Queue TCP ACK Upload" queue with the highest weight (100), which is part of the Upload Pipe that has CoDel enabled.

This seems to reduce latency of TCP handshakes when the Upload is heavily used.

I didn't measure anything though, I've just read at a few places that it would be a good choice to do.

As reference, here is my whole shaper config, I don't know how many forum posts I've read but I think it's pretty close to your docs, with more pipes and queues for better traffic shaping.

``` 10000 1 220 Mbit none fq_codel 1 5 100 1 0 1514 1000 1024 TrafficShaper Pipe 220 Mbits Download 10001 1 40 Mbit none fq_codel 1 5 100 1 0 1514 1000 1024 TrafficShaper Pipe 40 Mbits Upload 10000 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 90 none 0 0 0 Queue DNS Download TrafficShaper 10001 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 100 none 0 0 0 Queue TCP ACK Download TrafficShaper 10002 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 1 none 0 0 0 Queue Any Download TrafficShaper 10003 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 100 none 0 0 0 Queue TCP ACK Upload TrafficShaper 10004 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 90 none 0 0 0 Queue DNS Upload TrafficShaper 10005 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 1 none 0 0 0 Queue Any Upload TrafficShaper 10006 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 70 none 0 0 0 Queue SSH Upload TrafficShaper 10007 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 70 none 0 0 0 Queue SSH Download TrafficShaper 10008 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 20 none 0 0 0 Queue TCP Upload TrafficShaper 10009 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 20 none 0 0 0 Queue TCP Download TrafficShaper 10010 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 10 none 0 0 0 Queue UDP Download TrafficShaper 10011 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 10 none 0 0 0 Queue UDP Upload TrafficShaper 10012 1 6bcf9e9c-81c2-4e59-9649-339b35b1194e 80 none 0 0 0 Queue IPSEC Upload TrafficShaper 10013 1 bbd488dd-f2e9-4bed-8f9b-8004fc53b0e8 80 none 0 0 0 Queue IPSEC Download TrafficShaper 1 1 opt20 tcp_ack 100 any 0 any any 0 any out 2640ab36-2318-46d7-94e0-687ab2e45a73 TrafficShaper 1 2 opt20 tcp_ack 100 any 0 any any 0 any in 4c74bc97-7e86-4127-885b-fd8fcf14eece TrafficShaper 1 3 opt20 udp any 0 53 any 0 any out 7e0c9a63-5ddc-425a-8e31-15c084a50984 TrafficShaper 1 4 opt20 udp any 0 53 any 0 any in d523ae29-1925-4ba6-89ed-b6e4a0f9e120 TrafficShaper 1 5 opt20 udp any 0 4500 any 0 any out 98f3acf7-b46a-465e-a772-094fff00c27f TrafficShaper 1 6 opt20 udp any 0 4500 any 0 any in 801c81c2-49c4-493a-b164-a07203d2bb28 TrafficShaper 1 7 opt20 tcp any 0 22 any 0 any out 55294006-5142-4778-b830-642d3d4bb258 TrafficShaper 1 8 opt20 tcp any 0 22 any 0 any in e52619e0-e3cd-483a-b71f-ca975a4c8b9d TrafficShaper 1 9 opt20 tcp any 0 any any 0 any out 4198aac7-34bb-42a0-9899-85054fd2216d TrafficShaper 1 10 opt20 tcp any 0 any any 0 any in 7b28af34-2ca5-4830-9815-6ed365cdc697 TrafficShaper 1 11 opt20 udp any 0 any any 0 any out b1d15e66-4fb7-45bb-80e7-924167c4106b TrafficShaper 1 12 opt20 udp any 0 any any 0 any in ae7e74aa-df43-49c0-b806-14e348b6beb9 TrafficShaper 1 13 opt20 ip any 0 any any 0 any out 24410501-b253-4d14-82b5-c458250ee2ef TrafficShaper 1 14 opt20 ip any 0 any any 0 any in 1a62264f-e634-44b1-9c6a-b37d338f1daa TrafficShaper ```

[SeimusS]

I've read somewhere the same. However, I myself with the configuration I have, didn't yet seen/experienced latency on Upload during load, so its hard for me to test and write about it. However Download on my side is somewhat tricky (Due to my ISP's technological limitation)

I think the reason (in this specific case) why this Upload or Download latency may happen is due to the flows parameter, the default 1024 may be in some use-cases too low, which would cause some specific flows classified into same slots (sub-queues), this would cause for them as well a latency if that specific sub-queue is experience it.

Even if increase of flows parameters could improve it, ultimately I think creating a separate Queue for TCP ACK, which with CoDeL will create sub-queues for specific TCP ACK flows, is ultimately better way.

_This is just my rough (positive imagination) estimate, but usually the no-knobs deployment for FQ_CoDeL works in 98% of the time if the BW parameter in the Shaper Pipe is set properly. The 2% are then the setups were it needs tuning additional to the out of the box setup of FQCoDeL.

I may however maybe try it on Download Pipe to see if there will be improvements. If so we could added it as one of the tuning points into the guide.

I like your Shaper config, very diligent. Thank you for the config.

[mimugmail]

The problem is that there is no rule of thumb as it heavily depends on media type (LAN, Ethernet, DSL, Satelliten, Cable) and latency. E.g. for cable with variing upload speeds you may hit congestion earlier than your pipe kicks in

[SeimusS]

You are absolutely right, that's the main problem actually. For Download as well Upload alike.

For the Shaper/Fq_CoDeL to work all the time properly, you need to hit "persistently" the throughput specified in the BW parameter within the Pipe. Which on these technologies is hardly possible sometimes (or most of the time, really depends).

Even thou FQ_CoDeL does have mechanisms such as target & interval (another one is limit but... not usable currently) that measure the delay for each queue and can take action upon it, the fact that you set BW 500 but can reach at a time only 200M will cause a back-pressure which will result in increased latency.

_This specific I tested, and the results was even if this happens the latency is still lower with FQ_CoDel in such situation than without FQ_CoDeL at all. The margin in these situation was 50-120ms (with FQ_CoDel) vs 1200-2000ms (without FQCoDeL) during congestion time, worst case scenario.

As for the rule of thumb, there is none, but its recommended to target 85% of advertised ISP BW. Which in theory and most cases should include the worst case scenario throughput. However even this recommendation is recommended to take with grain of salt and needs to be tuned accordingly.

Thus doing few runs of Internet speed-tests, to possibly get average throughput and from it target 85% as the BW value in the Shaper is a good start. I would however extent this, to target 85% of the lowest possible throughput as your BW value, if somebody wants to always try prevention of bufferbloat.

The caveat is, the lowest possible that sometimes occurs maybe waaay to low thus you are loosing substantial amount of BW. Its a heavy tradeoff. For example if I have 500M and lowest throughput occurrence sometimes is 200M. I could set 85% of that 200M as the BW in Pipe. But I am cutting myself of 300M possible throughput that I am usually able to get just to prevent the occasional latency.

[SeimusS]

I did contact the original creators behind FQ_CoDeL and asked them to give potential review for explanation of FQ_CoDeL & its parameters. They gave me some feedback and tips.

I changed this PR to Draft. I will include their feedback into the How-To. As I want to have this as complete and perfect as possible so the wider audience is well informed, and has at least some grasp on what to configure as baseline and tune further if needed.

I am also retesting and checking further the configuration.

P.S. Also I found a mistake in the config example, the queue for object pipe shouldn't have any values (it has no configuration value), this is used for dynamic queue such as FiFo etc. when we are not specifying any Queue objects. Its not needed as we will create separate queue object binded to the specific Pipe.

Regards, S.

[SeimusS]

Few observations/notes from my side (OPNsense configuration):

A. CoDeL can be configured in Pipe

• If configured in Pipe, standalone Queue object that binds to the Pipe cant be used. Otherwise CoDel will not be applied on the PiPes dynamic queue
• Enabling the ECN option in Pipe will enable it as well for CoDeL
• CoDeL is applied to the whole dynamic Queue

B. CoDeL can be configured in Queue

• standalone Queue object bind to the Pipe
• Pipe setting CoDel even if checked has no configuration value when standalone Queue objects are configured
• Enabling the ECN option in Pipe does nothing
• Enabling the ECN option in Queue will enable it as well for CoDeL

C. FQ_CoDel is configured in scheduler

• Can be used with dynamic queues as well non dynamic Queues e.g., Create only Pipe with FQ_CoDeL or Pipe + Queue with FQ_CoDeL
• In theory each Flow is in its own Queue, classified by hashing the 5-tuple
• You can enable CoDeL in Queue, ipfw shows its configured (FQ_CodeL in scheduler CoDel in Queue) but doesn't bring any benefit
• Weights in Queues looks like do not have any impact, FQ_CoDeL is not honoring/using it
• If weighted Queues are needed combination of Codel AQM + (for example) WFQ Scheduler needs to be done
• MASK setting in Queues (same for Pipes) doesn't have any impact, When removed BW equality behaves the same
• Enabling the ECN option in Pipe will enable it as well for FQ_CoDeL in scheduler
• Enabling the ECN option in Queue does nothing (if CoDeL is disabled)

[SeimusS]

Hello @AdSchellevis

I think I am finished here. I was lucky enough to reach towards FQ_CoDel creators & bufferbloat community. They were kind enough to review fully the docs adjust it and provide more proper details about FQ_CoDel.

Special thanks to @richb-hanover & @dtaht for giving me their time, advice and full review!

P.S. last thing that is needed is just to check for grammar and spelling errors.

Regards, S.

[AdSchellevis]

@SeimusS very nice writeup, thanks for contributing it. I'll merge and push it to the docs.

[richb-hanover]

PS @SeimusS - you're famous(-ish)! https://randomneuronsfiring.com/opnsense-fights-bufferbloat/

Thanks again for letting me contribute.

[SeimusS]

@richb-hanover hahaha bit unexpected yet lovely of you. Many thanks for mentioning OPNsense (& its official docs for FQ_C) on your blog!

I had fun too.

Regards, S.

[SeimusS]

@AdSchellevis

One question I didn't put the reference for the doc "shaper_bufferbloat" into source/manual/shaping.rst its only in the source/manual/how-tos/shaper.rst.

Which makes it maybe a bit hidden when somebody looks at the Traffic Shaping manual as whole. Is it possible you will additionally put it there as well? Otherwise I can open another PR.

how-tos/shaper_bufferbloat

Regards, S.

[AdSchellevis]

@SeimusS I missed that as well, just added https://github.com/opnsense/docs/commit/39640f28abcc34892a14c8e034951f773c3ed288 which should help.

[richb-hanover]

@AdSchellevis @SeimusS Google Alerts for "bufferbloat" points me to this video:

https://www.youtube.com/watch?v=x05Nxw4E8fI

Enjoy!