search

opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.1k stars 702 forks source link

Add net.isr.maxthreads, net.isr.bindthreads, net.isr.dispatch to Tunables and adjust defaults #5415

Closed amezin closed 2 years ago

amezin commented 2 years ago

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

Is your feature request related to a problem? Please describe.

When running https://www.speedtest.net/ to my ISP's server, I noticed that OPNsense shows 20-30% packet loss on WAN.

It turns out that all PPP packets are handled on only one CPU core (with my hardware and default OPNsense configuration), and the single-core performance of E3950 is just not enough.

The issue (and the solution) is described in pfSense docs, FreeBSD Bugzilla

Describe the solution you like

I solved the issue by adding the following tunables:

1) I think OPNsense UI should show these tunables by default. 2) I think the default values for net.isr.maxthreads and net.isr.bindthreads should be adjusted. 3) Maybe OPNsense should show a warning on PPP configuration page.

net.isr.maxthreads: I think on a router/firewall it should be set to "all cores" (-1) by default. Also, as far as I understand, this tunable has no effect when net.isr.dispatch == direct (the default).

net.isr.bindthreads: I haven't tested its performance impact. But if a thread is created for every CPU core, I think it makes sense to bind every thread to its own core.

net.isr.maxthreads and net.isr.bindthreads can be set only on boot, so a good default value is a bit more important than usual.

net.isr.dispatch: I've read in multiple places that you should avoid changing it unless you absolutely need to. Still don't understand the details. But at least it can be changed without a reboot, so:

AdSchellevis commented 2 years ago

Optimal settings vary per situation (like enabling rss for example https://forum.opnsense.org/index.php?topic=24409.msg116941#msg116941).

Adding an optimal settings paragraph for ppp type interfaces in the documentation (https://github.com/opnsense/docs) might be a better option.

amezin commented 2 years ago

That forum thread you've mentioned also suggests setting net.isr.maxthreads=-1 and net.isr.bindthreads=1. And also:

If RSS is enabled with the 'enabled' sysctl, the packet dispatching policy will move from ‘direct’ to ‘hybrid’.

So if you're enabling rss you also effectively apply the configuration I've described here.

pfSense docs suggest that they have net.isr.maxthreads=-1 as the default too (not sure about bindthreads):

Tuning the values of net.isr.maxthreads and net.isr.numthreads may yield additional performance gains. Generally these are best left at default values matching the number of CPU cores, but depending on the workload may work better at lower values.

And, again, maxthreads and bindthreads don't seem to have any effect unless you also enable deferred/hybrid dispatch or rss (or maybe something else) - i. e. if there are no threads.

So why not change the default? (I'm not asking for net.isr.dispatch, since that could cause regressions - only for maxthreads and bindthreads)

fichtner commented 2 years ago

I‘m sure you can make this easier by sharing a comparison table across hardware with throughput and sysctl combinations. It’ll make changing the defaults that much easier. Thanks!

lordraiden commented 2 years ago

I‘m sure you can make this easier by sharing a comparison table across hardware with throughput and sysctl combinations. It’ll make changing the defaults that much easier. Thanks!

Or just use Google to find the evidence of a topic that has been beaten to death and documented in multiple forums

lordraiden commented 2 years ago

https://redmine.pfsense.org/issues/4821

And the golden source https://calomel.org/freebsd_network_tuning.html Old but good https://calomel.org/network_performance.html

fichtner commented 2 years ago

Yeah, not really. ;)

lordraiden commented 2 years ago

Yeah, not really. ;)

Yeah

fichtner commented 2 years ago

So what’s the reason FreeBSD has this default and doesn’t listen to “authorities” on the subject?

lordraiden commented 2 years ago

So what’s the reason FreeBSD has this default and doesn’t listen to “authorities” on the subject?

Maybe is because freebsd is not designed to be a firewall only, I don't know

Men you asked for documentation (all the settings are explained with pro and cons), evidence and tests, now you have it, do as you wish.

If I was rude in my first comment accept my apologies

lordraiden commented 2 years ago

In addition @amezin was asking for exposing the settings in the web ui if instead of making them the default if you see any risks. I think this will help people but you have to value how important is. 85% of lines in my country (all but 1 operator) are using ppoe.

If you decide to review the information there are more settings that can be exposed and will help performance

mimugmail commented 2 years ago

@lordraiden its nearly impossible to find something stable for all situations and scenarios with just google and since operating System changes defaults over time (and CPU gets better) its also quite hard to keep defaults up2date. Similar to a discussion about IPsec defaults (which is way easier to handle). Usually you only change defaults when know 100% the impact. IMHO there needs to be a reproducable testbed and compare speedtests with every tunable showing the differences, then put this in docs (like pf does this too).

lordraiden commented 2 years ago

@mimugmail then expose the settings in the UI explain them base on the config file commented by calomel and let the people choose.

Some settings are validated and documented by other comercial firewalls so you are not jumping into the void

mimugmail commented 2 years ago

It is already, you add a tunable via UI?! The rest is up to the docs.

mimugmail commented 2 years ago

BTW, how lucky you are, dont get me wrong, in Germany also most of the users have pppoe, but speeds at 100 or 250 mbit are still very high. Here at home I have a 100/50 fiber pppoe conneted with a standard slow Celeron: CPU: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz (2000.05-MHz K8-class CPU) This is my speedtest:

root@mimu-fw:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (80.151.56.135)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by FoxHost (Falkenstein) [83.95 km]: 22.812 ms
Testing download speed................................................................................
Download: 104.00 Mbit/s
Testing upload speed......................................................................................................
Upload: 53.62 Mbit/s

Quite sure also @fichtner has same values and Deciso in Netherlands don't use pppoe. Really, dont get me wrong, it needs qualified testing when changing defaults, referencing to a site, no matter about it's reputation shouldn't let anyone changing such things.

I'm also sure most commercial firewall vendors won't let you change such things as this would lead to very high support rates as ppl would change things read via google results turning into desaster.

I remember long time ago where I had a warning about a full disk in a central firewall manager. In hurry a googled and first hit was to reinit the database. Blindly executing this command wiped the db and also send all firewalls an empty config :) (wasnt one of my best days I guess)

lordraiden commented 2 years ago

In my country everyone but Orange uses pppoe. 1gb connection are available in any big city or town although the standard is 300/600 mbps. Some providers like Digi have already customers with 10gb as a beta service and the ont and router provided by orange when you buy the 1gbps service is already 2.5 gbps capable

mimugmail commented 2 years ago

Then most of the core dev's are not able to reproduce which makes it even harder to change defaults.

IMHO the best way would be:

It should be easy to see which knobs would gain more throughput and at first can be added to the docs. Then let this evolve over time, let other with less throughput or evern other WAN uplink use against these values. Then, at some time, defaults can be changed.

Just my 2 cents :)

L1ghtn1ng commented 2 years ago

Just to add the UK uses PPPoE as well, it is more common than you think. Would be an idea to ask people on twitter maybe and in the forum and point people to a survey to get better understanding maybe? Or as @amezin mentioned detect that PPPoE is used, when you set your wan interface to it and then apply the changes in question? Mind you not sure what these changes would do if you are using RSS though. Just some food for thought, but docs for this would be a good start and if you pick PPPoE for your WAN interface, could have some help text in the UI that says you might want to check this url to the docs page in question to help get better throughput with this connection type you have as well which would be good.

OPNsense-bot commented 2 years ago

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.

brad0 commented 1 year ago

Just to add the UK uses PPPoE as well, it is more common than you think.

In Canada as well with Bell Canada with GPON / XGS-PON too.

Zerophase commented 1 month ago

I did not formally bench it. But, when I was tuning my Dec850 for throughput turning these three settings on seems to significantly improve boot speeds.