search

Ysurac / openmptcprouter

OpenMPTCProuter is an open source solution to aggregate multiple internet connections using Multipath TCP (MPTCP) on OpenWrt
https://www.openmptcprouter.com/
GNU General Public License v3.0
1.78k stars 257 forks source link

Download speed issue #104

Closed SoGooDFR closed 6 years ago

SoGooDFR commented 6 years ago

Expected Behavior

Get full of my speed connection in dowload or youtube, stream...

Actual Behavior

I can get my full speed connection in a speedtest (80-100mbps) I get 4-6 mbps on youtube. I get 4-6 mbps on twitch.

I get only 4-6mpbs when i downlad file from http://ovh.net/files or http://ping.online.net...

Steps to Reproduce the Problem

  1. Fresh install on both
  2. Enter the basic configuration
  3. See the results.

Specifications

Ysurac commented 6 years ago

From what platform do you download file ? what file do you download ? what is the command line used ? How do you check speed on youtube ?

SoGooDFR commented 6 years ago

Some files.dat. On youtube, right-click and select "Advanced Statistics". I check the speed connection of the VPS with iperf = 1-3G/s.

fnoir commented 6 years ago

I have same problem. OpenMPTCProuter use only the master connection for download :(

Ysurac commented 6 years ago

Without info, I can't reproduce and fix anything...

I need:

fnoir commented 6 years ago

Sorry,

I use Raspberry Pi 3 Model B Rev 1.2 (openmptcprouter v0.27 r7308-ebe1216c7c) for router at home

Ysurac commented 6 years ago

Can you run, on OpenMPTCProuter, iperf3 -c bouygues.iperf.fr -P 10 -4 and give me the result of the last two SUM line ? Also tell me the speed of your links. What are the ADSL and 4G FAI ?

fnoir commented 6 years ago 
[SUM]   0.00-10.00  sec   110 MBytes  92.6 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec  22.2 MBytes  18.6 Mbits/sec                  receiver

Free for both

SoGooDFR commented 6 years ago

For me :

Iperf : 

[SUM]   0.00-10.00  sec   110 MBytes  92.3 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec  31.6 MBytes  26.5 Mbits/sec                  receiver

Edit: Same bandwidth limitation with Shadowsocks disabled and only Glorytun UDP.

Ysurac commented 6 years ago

What is the result of iperf3 -c bouygues.iperf.fr -P 10 -4 -R ?

fnoir commented 6 years ago 
[SUM]   0.00-10.00  sec  53.8 MBytes  45.1 Mbits/sec    3             sender
[SUM]   0.00-10.00  sec  31.9 MBytes  26.8 Mbits/sec                  receiver
SoGooDFR commented 6 years ago 
[SUM]   0.00-10.00  sec  56.6 MBytes  47.5 Mbits/sec  171             sender
[SUM]   0.00-10.00  sec  47.1 MBytes  39.5 Mbits/sec                  receiver
fnoir commented 6 years ago

I retry this morning : 

[SUM]   0.00-10.00  sec  73.7 MBytes  61.8 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec  54.3 MBytes  45.5 Mbits/sec                  receiver
boundouny commented 6 years ago

I don't know if this is can helf, but I found a same problem in overthebox's user, it's a loadbalancing ping problem ... If you want i can send an URL where somme people have the same problem and the solution...

they always have same configuration: 1 x 4G and 1 x ADSL conection

nileio commented 6 years ago

i am facing the same problem as described above , and i still have no solution, i am in a similar config to @SoGooDFR and @fnoir . I am following this thread ! what is the solution you refer to @boundouny ?

Ysurac commented 6 years ago

If you set Multipath TCP SYN retries to 5 in Network->MPTCP, this should give a little bit more usable speed in some cases. You can also try to disable Multipath TCP checksum.

SoGooDFR commented 6 years ago

@boundouny But i only use one connection : 4G, so we don't have any load balancing here.

fnoir commented 6 years ago

I try solutions but not working.

When i test my connection with speedtest.net => 25Mbps, i download a file with ovh.net/files/ =>12Mbps

Ysurac commented 6 years ago

What file do you download and what is the command used ?

fnoir commented 6 years ago

I use curl http://ovh.net/files/1Gio.dat --output 1Gio.dat => Current speed: 1445k

Speedtest: speedtest

Ysurac commented 6 years ago

You should use: curl http://ovh.net/files/1Gio.dat -o /dev/null

fnoir commented 6 years ago

On Windows, i use curl http://ovh.net/files/1Gio.dat -o nul => Current speed: 1398k

boundouny commented 6 years ago

@SoGooDFR I speak about this: https://community.ovh.com/t/perte-debit-4g-avec-otb/8498/7 and https://community.ovh.com/t/perte-de-lupload-de-la-box-4g-bouygues/3490/5

catslab commented 6 years ago

speedtest.net is not accurate at all. It shows me 4Mbps upload where I can only have 2 max. Try nperf.com if you need web based test.

SoGooDFR commented 6 years ago

@catslab Ok, but for now, this is change nothing about the problem. With my 4G, i can get 120 mbps the night, but on navigation (stream, video, download,) i'm limited at 5-6 mpbs...

openmptcp_lol

fnoir commented 6 years ago

@catslab Same result with nperf.com

Download: 51.73 Mbps (av: 44.72) Upload: 16.98 Mbps (av: 13.98) Ping: 24 ms

Ysurac commented 6 years ago

I think I find the issue: sysctl -w net.ipv4.tcp_moderate_rcvbuf=1 It was set to 0. This will be fixed in next release that should be out tomorrow.

SoGooDFR commented 6 years ago

Thx man but i test your temporary fix and again in a speedtest it give to me 70-80 mbps. On navigation, i'm stuck with 20-25 mbps. I can see a littre improvement, so the bug/problem persist.

Ysurac commented 6 years ago

After this change all networks services must be restarted (it's also possible to remove the line in /etc/sysctl.d/default.conf).

Without reproducible examples I can't test and fix anything. What is the test for "On navigation" ?

SoGooDFR commented 6 years ago

I apply the line fix, reload the sysctl config on modem and see the results. Navigation test :

For all, i will get the same limitation : 25 Mbps.

Ysurac commented 6 years ago

When I ask about reproducible examples, I mean command line used and URL. Also YouTube is not a good speed test.

SoGooDFR commented 6 years ago

Pls, stop blind your eyes as OVH. My test is as end-user test. Command line is just the first line test to see a revelant problem or not. It's important to not only based conclusion on a commande line...

So this is the result : 

curl http://ovh.net/files/1Gio.dat -o nul
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  6 1024M    6 63.9M    0     0  2111k      0  0:08:16  0:00:31  0:07:45 2457k^C
Ysurac commented 6 years ago

How you want me to check where is the issue if I can't reproduce it ? I can just ignore you but I try to find the issue. And again if you want me to check where is the issue, I need the command line used, the URL used,... I need to see it to fix it.

SoGooDFR commented 6 years ago

I edit and give you the command in my last post. In fact, i make many posts to explain to you (and other people) that we don't have limited speed only on speedtest (nperf, speedtest, fast) (command, browser).

fnoir commented 6 years ago

@SoGooDFR @Ysurac What's your 4G ISP ?

@Ysurac If i give you a remote access to my LAN, can you find problem ?

SoGooDFR commented 6 years ago

@fnoir The red cube.

fnoir commented 6 years ago

@SoGooDFR thanks and your router ? i use Huawei B593

SoGooDFR commented 6 years ago

@fnoir B525-23a

SoGooDFR commented 6 years ago

@Ysurac At the same time i make same tests with my unique 4G connection on OTB also with ShadowSocks. I get my full speed.

Ysurac commented 6 years ago

Could you give me the result of sysctl -a on OTB ?

SoGooDFR commented 6 years ago 
 PACKAGE:     otb
 VERSION:     v3.4

 BUILD REPO:  https://github.com/ovh/overthebox
 BUILD DATE:  Thu Mar  1 09:44:08 UTC 2018
-----------------------------------------------------
root@OverTheBox:~# sysctl -a
debug.exception-trace = 1
dev.hpet.max-user-freq = 64
dev.scsi.logging_level = 0
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.dentry-state = 7514  4967    45      0       0       0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 208117
fs.file-max = 51200
fs.file-nr = 512        0       51200
fs.inode-nr = 5989      0
fs.inode-state = 5989   0       0       0       0       0       0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mount-max = 100000
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.pipe-user-pages-hard = 0
fs.pipe-user-pages-soft = 16384
fs.protected_hardlinks = 0
fs.protected_symlinks = 0
fs.suid_dumpable = 2
kernel.auto_msgmni = 0
kernel.bootloader_type = 33
kernel.bootloader_version = 1
kernel.cad_pid = 1
kernel.cap_last_cap = 37
kernel.core_pattern = /tmp/%e.%t.%p.%s.core
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 0
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = (none)
kernel.hostname = OverTheBox
kernel.hotplug = /sbin/hotplug
kernel.io_delay_type = 0
kernel.kexec_load_disabled = 0
kernel.kptr_restrict = 0
kernel.kstack_depth_to_print = 12
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 32000
kernel.ngroups_max = 65536
kernel.osrelease = 4.9.58
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 3
kernel.panic_on_io_nmi = 0
kernel.panic_on_oops = 1
kernel.panic_on_rcu_stall = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.panic_on_warn = 0
kernel.perf_cpu_time_max_percent = 25
kernel.perf_event_max_contexts_per_stack = 8
kernel.perf_event_max_sample_rate = 100000
kernel.perf_event_max_stack = 127
kernel.perf_event_mlock_kb = 516
kernel.perf_event_paranoid = 2
kernel.pid_max = 32768
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 7       4       1       7
kernel.printk_delay = 0
kernel.printk_devkmsg = ratelimit
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 1
kernel.pty.reserve = 1024
kernel.random.boot_id = 35359d4d-4aeb-44b2-ba8a-4c330116b6b9
kernel.random.entropy_avail = 3177
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.urandom_min_reseed_secs = 60
kernel.random.uuid = b7a73edb-406e-423a-a1b6-796d5ff731bc
kernel.random.write_wakeup_threshold = 3072
kernel.randomize_va_space = 2
kernel.real-root-dev = 0
kernel.sched_child_runs_first = 0
kernel.sched_rr_timeslice_ms = 10
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sem = 32000      1024000000      500     32000
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
kernel.sysctl_writes_strict = 1
kernel.tainted = 0
kernel.threads-max = 7690
kernel.unknown_nmi_panic = 0
kernel.unprivileged_bpf_disabled = 0
kernel.usermodehelper.bset = 4294967295 63
kernel.usermodehelper.inheritable = 4294967295  63
kernel.version = #0 SMP Fri Nov 3 10:31:39 2017
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.default_qdisc = fq_codel
net.core.dev_weight = 64
net.core.flow_limit_cpu_bitmap = 0
net.core.flow_limit_table_len = 4096
net.core.max_skb_frags = 17
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_max_backlog = 4096
net.core.netdev_rss_key = 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 65536
net.core.rmem_max = 134217728
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 4096
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 65536
net.core.wmem_max = 134217728
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.drop_gratuitous_arp = 0
net.ipv4.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.all.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.all.ignore_routes_with_linkdown = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.route_localnet = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.drop_gratuitous_arp = 0
net.ipv4.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.default.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.default.ignore_routes_with_linkdown = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.route_localnet = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.drop_gratuitous_arp = 0
net.ipv4.conf.eth0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.eth0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.route_localnet = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.if0.accept_local = 0
net.ipv4.conf.if0.accept_redirects = 1
net.ipv4.conf.if0.accept_source_route = 1
net.ipv4.conf.if0.arp_accept = 0
net.ipv4.conf.if0.arp_announce = 0
net.ipv4.conf.if0.arp_filter = 0
net.ipv4.conf.if0.arp_ignore = 1
net.ipv4.conf.if0.arp_notify = 0
net.ipv4.conf.if0.bootp_relay = 0
net.ipv4.conf.if0.disable_policy = 0
net.ipv4.conf.if0.disable_xfrm = 0
net.ipv4.conf.if0.drop_gratuitous_arp = 0
net.ipv4.conf.if0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.if0.force_igmp_version = 0
net.ipv4.conf.if0.forwarding = 1
net.ipv4.conf.if0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.if0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.if0.ignore_routes_with_linkdown = 0
net.ipv4.conf.if0.log_martians = 0
net.ipv4.conf.if0.mc_forwarding = 0
net.ipv4.conf.if0.medium_id = 0
net.ipv4.conf.if0.promote_secondaries = 0
net.ipv4.conf.if0.proxy_arp = 0
net.ipv4.conf.if0.proxy_arp_pvlan = 0
net.ipv4.conf.if0.route_localnet = 0
net.ipv4.conf.if0.rp_filter = 0
net.ipv4.conf.if0.secure_redirects = 1
net.ipv4.conf.if0.send_redirects = 1
net.ipv4.conf.if0.shared_media = 1
net.ipv4.conf.if0.src_valid_mark = 0
net.ipv4.conf.if0.tag = 0
net.ipv4.conf.if1.accept_local = 0
net.ipv4.conf.if1.accept_redirects = 1
net.ipv4.conf.if1.accept_source_route = 1
net.ipv4.conf.if1.arp_accept = 0
net.ipv4.conf.if1.arp_announce = 0
net.ipv4.conf.if1.arp_filter = 0
net.ipv4.conf.if1.arp_ignore = 1
net.ipv4.conf.if1.arp_notify = 0
net.ipv4.conf.if1.bootp_relay = 0
net.ipv4.conf.if1.disable_policy = 0
net.ipv4.conf.if1.disable_xfrm = 0
net.ipv4.conf.if1.drop_gratuitous_arp = 0
net.ipv4.conf.if1.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.if1.force_igmp_version = 0
net.ipv4.conf.if1.forwarding = 1
net.ipv4.conf.if1.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.if1.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.if1.ignore_routes_with_linkdown = 0
net.ipv4.conf.if1.log_martians = 0
net.ipv4.conf.if1.mc_forwarding = 0
net.ipv4.conf.if1.medium_id = 0
net.ipv4.conf.if1.promote_secondaries = 0
net.ipv4.conf.if1.proxy_arp = 0
net.ipv4.conf.if1.proxy_arp_pvlan = 0
net.ipv4.conf.if1.route_localnet = 0
net.ipv4.conf.if1.rp_filter = 0
net.ipv4.conf.if1.secure_redirects = 1
net.ipv4.conf.if1.send_redirects = 1
net.ipv4.conf.if1.shared_media = 1
net.ipv4.conf.if1.src_valid_mark = 0
net.ipv4.conf.if1.tag = 0
net.ipv4.conf.ifb0.accept_local = 0
net.ipv4.conf.ifb0.accept_redirects = 1
net.ipv4.conf.ifb0.accept_source_route = 1
net.ipv4.conf.ifb0.arp_accept = 0
net.ipv4.conf.ifb0.arp_announce = 0
net.ipv4.conf.ifb0.arp_filter = 0
net.ipv4.conf.ifb0.arp_ignore = 1
net.ipv4.conf.ifb0.arp_notify = 0
net.ipv4.conf.ifb0.bootp_relay = 0
net.ipv4.conf.ifb0.disable_policy = 0
net.ipv4.conf.ifb0.disable_xfrm = 0
net.ipv4.conf.ifb0.drop_gratuitous_arp = 0
net.ipv4.conf.ifb0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.ifb0.force_igmp_version = 0
net.ipv4.conf.ifb0.forwarding = 1
net.ipv4.conf.ifb0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.ifb0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.ifb0.ignore_routes_with_linkdown = 0
net.ipv4.conf.ifb0.log_martians = 0
net.ipv4.conf.ifb0.mc_forwarding = 0
net.ipv4.conf.ifb0.medium_id = 0
net.ipv4.conf.ifb0.promote_secondaries = 0
net.ipv4.conf.ifb0.proxy_arp = 0
net.ipv4.conf.ifb0.proxy_arp_pvlan = 0
net.ipv4.conf.ifb0.route_localnet = 0
net.ipv4.conf.ifb0.rp_filter = 0
net.ipv4.conf.ifb0.secure_redirects = 1
net.ipv4.conf.ifb0.send_redirects = 1
net.ipv4.conf.ifb0.shared_media = 1
net.ipv4.conf.ifb0.src_valid_mark = 0
net.ipv4.conf.ifb0.tag = 0
net.ipv4.conf.ifb1.accept_local = 0
net.ipv4.conf.ifb1.accept_redirects = 1
net.ipv4.conf.ifb1.accept_source_route = 1
net.ipv4.conf.ifb1.arp_accept = 0
net.ipv4.conf.ifb1.arp_announce = 0
net.ipv4.conf.ifb1.arp_filter = 0
net.ipv4.conf.ifb1.arp_ignore = 1
net.ipv4.conf.ifb1.arp_notify = 0
net.ipv4.conf.ifb1.bootp_relay = 0
net.ipv4.conf.ifb1.disable_policy = 0
net.ipv4.conf.ifb1.disable_xfrm = 0
net.ipv4.conf.ifb1.drop_gratuitous_arp = 0
net.ipv4.conf.ifb1.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.ifb1.force_igmp_version = 0
net.ipv4.conf.ifb1.forwarding = 1
net.ipv4.conf.ifb1.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.ifb1.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.ifb1.ignore_routes_with_linkdown = 0
net.ipv4.conf.ifb1.log_martians = 0
net.ipv4.conf.ifb1.mc_forwarding = 0
net.ipv4.conf.ifb1.medium_id = 0
net.ipv4.conf.ifb1.promote_secondaries = 0
net.ipv4.conf.ifb1.proxy_arp = 0
net.ipv4.conf.ifb1.proxy_arp_pvlan = 0
net.ipv4.conf.ifb1.route_localnet = 0
net.ipv4.conf.ifb1.rp_filter = 0
net.ipv4.conf.ifb1.secure_redirects = 1
net.ipv4.conf.ifb1.send_redirects = 1
net.ipv4.conf.ifb1.shared_media = 1
net.ipv4.conf.ifb1.src_valid_mark = 0
net.ipv4.conf.ifb1.tag = 0
net.ipv4.conf.lan.accept_local = 0
net.ipv4.conf.lan.accept_redirects = 1
net.ipv4.conf.lan.accept_source_route = 1
net.ipv4.conf.lan.arp_accept = 0
net.ipv4.conf.lan.arp_announce = 0
net.ipv4.conf.lan.arp_filter = 0
net.ipv4.conf.lan.arp_ignore = 1
net.ipv4.conf.lan.arp_notify = 0
net.ipv4.conf.lan.bootp_relay = 0
net.ipv4.conf.lan.disable_policy = 0
net.ipv4.conf.lan.disable_xfrm = 0
net.ipv4.conf.lan.drop_gratuitous_arp = 0
net.ipv4.conf.lan.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.lan.force_igmp_version = 0
net.ipv4.conf.lan.forwarding = 1
net.ipv4.conf.lan.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.lan.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.lan.ignore_routes_with_linkdown = 0
net.ipv4.conf.lan.log_martians = 0
net.ipv4.conf.lan.mc_forwarding = 0
net.ipv4.conf.lan.medium_id = 0
net.ipv4.conf.lan.promote_secondaries = 0
net.ipv4.conf.lan.proxy_arp = 0
net.ipv4.conf.lan.proxy_arp_pvlan = 0
net.ipv4.conf.lan.route_localnet = 0
net.ipv4.conf.lan.rp_filter = 0
net.ipv4.conf.lan.secure_redirects = 1
net.ipv4.conf.lan.send_redirects = 1
net.ipv4.conf.lan.shared_media = 1
net.ipv4.conf.lan.src_valid_mark = 0
net.ipv4.conf.lan.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.drop_gratuitous_arp = 0
net.ipv4.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.lo.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.lo.ignore_routes_with_linkdown = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.route_localnet = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.conf.tun0.accept_local = 0
net.ipv4.conf.tun0.accept_redirects = 1
net.ipv4.conf.tun0.accept_source_route = 1
net.ipv4.conf.tun0.arp_accept = 0
net.ipv4.conf.tun0.arp_announce = 0
net.ipv4.conf.tun0.arp_filter = 0
net.ipv4.conf.tun0.arp_ignore = 1
net.ipv4.conf.tun0.arp_notify = 0
net.ipv4.conf.tun0.bootp_relay = 0
net.ipv4.conf.tun0.disable_policy = 0
net.ipv4.conf.tun0.disable_xfrm = 0
net.ipv4.conf.tun0.drop_gratuitous_arp = 0
net.ipv4.conf.tun0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.tun0.force_igmp_version = 0
net.ipv4.conf.tun0.forwarding = 1
net.ipv4.conf.tun0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.tun0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.tun0.ignore_routes_with_linkdown = 0
net.ipv4.conf.tun0.log_martians = 0
net.ipv4.conf.tun0.mc_forwarding = 0
net.ipv4.conf.tun0.medium_id = 0
net.ipv4.conf.tun0.promote_secondaries = 0
net.ipv4.conf.tun0.proxy_arp = 0
net.ipv4.conf.tun0.proxy_arp_pvlan = 0
net.ipv4.conf.tun0.route_localnet = 0
net.ipv4.conf.tun0.rp_filter = 0
net.ipv4.conf.tun0.secure_redirects = 1
net.ipv4.conf.tun0.send_redirects = 1
net.ipv4.conf.tun0.shared_media = 1
net.ipv4.conf.tun0.src_valid_mark = 0
net.ipv4.conf.tun0.tag = 0
net.ipv4.conf.xtun0.accept_local = 0
net.ipv4.conf.xtun0.accept_redirects = 1
net.ipv4.conf.xtun0.accept_source_route = 1
net.ipv4.conf.xtun0.arp_accept = 0
net.ipv4.conf.xtun0.arp_announce = 0
net.ipv4.conf.xtun0.arp_filter = 0
net.ipv4.conf.xtun0.arp_ignore = 1
net.ipv4.conf.xtun0.arp_notify = 0
net.ipv4.conf.xtun0.bootp_relay = 0
net.ipv4.conf.xtun0.disable_policy = 0
net.ipv4.conf.xtun0.disable_xfrm = 0
net.ipv4.conf.xtun0.drop_gratuitous_arp = 0
net.ipv4.conf.xtun0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.xtun0.force_igmp_version = 0
net.ipv4.conf.xtun0.forwarding = 1
net.ipv4.conf.xtun0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.xtun0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.xtun0.ignore_routes_with_linkdown = 0
net.ipv4.conf.xtun0.log_martians = 0
net.ipv4.conf.xtun0.mc_forwarding = 0
net.ipv4.conf.xtun0.medium_id = 0
net.ipv4.conf.xtun0.promote_secondaries = 0
net.ipv4.conf.xtun0.proxy_arp = 0
net.ipv4.conf.xtun0.proxy_arp_pvlan = 0
net.ipv4.conf.xtun0.route_localnet = 0
net.ipv4.conf.xtun0.rp_filter = 0
net.ipv4.conf.xtun0.secure_redirects = 1
net.ipv4.conf.xtun0.send_redirects = 1
net.ipv4.conf.xtun0.shared_media = 1
net.ipv4.conf.xtun0.src_valid_mark = 0
net.ipv4.conf.xtun0.tag = 0
net.ipv4.fib_multipath_use_neigh = 0
net.ipv4.fwmark_reflect = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_msgs_burst = 50
net.ipv4.icmp_msgs_per_sec = 1000
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_link_local_mcast_reports = 1
net.ipv4.igmp_max_memberships = 100
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_qrv = 2
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_early_demux = 1
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_local_port_range = 10000    65000
net.ipv4.ip_local_reserved_ports =
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 0
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time = 30
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_resolicit = 0
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time = 100
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 30
net.ipv4.neigh.default.unres_qlen_bytes = 65536
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time = 30
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_resolicit = 0
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time = 100
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 30
net.ipv4.neigh.eth0.unres_qlen_bytes = 65536
net.ipv4.neigh.if0.anycast_delay = 100
net.ipv4.neigh.if0.app_solicit = 0
net.ipv4.neigh.if0.base_reachable_time = 30
net.ipv4.neigh.if0.base_reachable_time_ms = 30000
net.ipv4.neigh.if0.delay_first_probe_time = 5
net.ipv4.neigh.if0.gc_stale_time = 60
net.ipv4.neigh.if0.locktime = 100
net.ipv4.neigh.if0.mcast_resolicit = 0
net.ipv4.neigh.if0.mcast_solicit = 3
net.ipv4.neigh.if0.proxy_delay = 80
net.ipv4.neigh.if0.proxy_qlen = 64
net.ipv4.neigh.if0.retrans_time = 100
net.ipv4.neigh.if0.retrans_time_ms = 1000
net.ipv4.neigh.if0.ucast_solicit = 3
net.ipv4.neigh.if0.unres_qlen = 30
net.ipv4.neigh.if0.unres_qlen_bytes = 65536
net.ipv4.neigh.if1.anycast_delay = 100
net.ipv4.neigh.if1.app_solicit = 0
net.ipv4.neigh.if1.base_reachable_time = 30
net.ipv4.neigh.if1.base_reachable_time_ms = 30000
net.ipv4.neigh.if1.delay_first_probe_time = 5
net.ipv4.neigh.if1.gc_stale_time = 60
net.ipv4.neigh.if1.locktime = 100
net.ipv4.neigh.if1.mcast_resolicit = 0
net.ipv4.neigh.if1.mcast_solicit = 3
net.ipv4.neigh.if1.proxy_delay = 80
net.ipv4.neigh.if1.proxy_qlen = 64
net.ipv4.neigh.if1.retrans_time = 100
net.ipv4.neigh.if1.retrans_time_ms = 1000
net.ipv4.neigh.if1.ucast_solicit = 3
net.ipv4.neigh.if1.unres_qlen = 30
net.ipv4.neigh.if1.unres_qlen_bytes = 65536
net.ipv4.neigh.ifb0.anycast_delay = 100
net.ipv4.neigh.ifb0.app_solicit = 0
net.ipv4.neigh.ifb0.base_reachable_time = 30
net.ipv4.neigh.ifb0.base_reachable_time_ms = 30000
net.ipv4.neigh.ifb0.delay_first_probe_time = 5
net.ipv4.neigh.ifb0.gc_stale_time = 60
net.ipv4.neigh.ifb0.locktime = 100
net.ipv4.neigh.ifb0.mcast_resolicit = 0
net.ipv4.neigh.ifb0.mcast_solicit = 3
net.ipv4.neigh.ifb0.proxy_delay = 80
net.ipv4.neigh.ifb0.proxy_qlen = 64
net.ipv4.neigh.ifb0.retrans_time = 100
net.ipv4.neigh.ifb0.retrans_time_ms = 1000
net.ipv4.neigh.ifb0.ucast_solicit = 3
net.ipv4.neigh.ifb0.unres_qlen = 30
net.ipv4.neigh.ifb0.unres_qlen_bytes = 65536
net.ipv4.neigh.ifb1.anycast_delay = 100
net.ipv4.neigh.ifb1.app_solicit = 0
net.ipv4.neigh.ifb1.base_reachable_time = 30
net.ipv4.neigh.ifb1.base_reachable_time_ms = 30000
net.ipv4.neigh.ifb1.delay_first_probe_time = 5
net.ipv4.neigh.ifb1.gc_stale_time = 60
net.ipv4.neigh.ifb1.locktime = 100
net.ipv4.neigh.ifb1.mcast_resolicit = 0
net.ipv4.neigh.ifb1.mcast_solicit = 3
net.ipv4.neigh.ifb1.proxy_delay = 80
net.ipv4.neigh.ifb1.proxy_qlen = 64
net.ipv4.neigh.ifb1.retrans_time = 100
net.ipv4.neigh.ifb1.retrans_time_ms = 1000
net.ipv4.neigh.ifb1.ucast_solicit = 3
net.ipv4.neigh.ifb1.unres_qlen = 30
net.ipv4.neigh.ifb1.unres_qlen_bytes = 65536
net.ipv4.neigh.lan.anycast_delay = 100
net.ipv4.neigh.lan.app_solicit = 0
net.ipv4.neigh.lan.base_reachable_time = 30
net.ipv4.neigh.lan.base_reachable_time_ms = 30000
net.ipv4.neigh.lan.delay_first_probe_time = 5
net.ipv4.neigh.lan.gc_stale_time = 60
net.ipv4.neigh.lan.locktime = 100
net.ipv4.neigh.lan.mcast_resolicit = 0
net.ipv4.neigh.lan.mcast_solicit = 3
net.ipv4.neigh.lan.proxy_delay = 80
net.ipv4.neigh.lan.proxy_qlen = 64
net.ipv4.neigh.lan.retrans_time = 100
net.ipv4.neigh.lan.retrans_time_ms = 1000
net.ipv4.neigh.lan.ucast_solicit = 3
net.ipv4.neigh.lan.unres_qlen = 30
net.ipv4.neigh.lan.unres_qlen_bytes = 65536
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time = 30
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_resolicit = 0
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time = 100
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 30
net.ipv4.neigh.lo.unres_qlen_bytes = 65536
net.ipv4.neigh.tun0.anycast_delay = 100
net.ipv4.neigh.tun0.app_solicit = 0
net.ipv4.neigh.tun0.base_reachable_time = 30
net.ipv4.neigh.tun0.base_reachable_time_ms = 30000
net.ipv4.neigh.tun0.delay_first_probe_time = 5
net.ipv4.neigh.tun0.gc_stale_time = 60
net.ipv4.neigh.tun0.locktime = 100
net.ipv4.neigh.tun0.mcast_resolicit = 0
net.ipv4.neigh.tun0.mcast_solicit = 3
net.ipv4.neigh.tun0.proxy_delay = 80
net.ipv4.neigh.tun0.proxy_qlen = 64
net.ipv4.neigh.tun0.retrans_time = 100
net.ipv4.neigh.tun0.retrans_time_ms = 1000
net.ipv4.neigh.tun0.ucast_solicit = 3
net.ipv4.neigh.tun0.unres_qlen = 30
net.ipv4.neigh.tun0.unres_qlen_bytes = 65536
net.ipv4.neigh.xtun0.anycast_delay = 100
net.ipv4.neigh.xtun0.app_solicit = 0
net.ipv4.neigh.xtun0.base_reachable_time = 30
net.ipv4.neigh.xtun0.base_reachable_time_ms = 30000
net.ipv4.neigh.xtun0.delay_first_probe_time = 5
net.ipv4.neigh.xtun0.gc_stale_time = 60
net.ipv4.neigh.xtun0.locktime = 100
net.ipv4.neigh.xtun0.mcast_resolicit = 0
net.ipv4.neigh.xtun0.mcast_solicit = 3
net.ipv4.neigh.xtun0.proxy_delay = 80
net.ipv4.neigh.xtun0.proxy_qlen = 64
net.ipv4.neigh.xtun0.retrans_time = 100
net.ipv4.neigh.xtun0.retrans_time_ms = 1000
net.ipv4.neigh.xtun0.ucast_solicit = 3
net.ipv4.neigh.xtun0.unres_qlen = 30
net.ipv4.neigh.xtun0.unres_qlen_bytes = 65536
net.ipv4.ping_group_range = 1   0
net.ipv4.route.error_burst = 500
net.ipv4.route.error_cost = 100
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = -1
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 2147483647
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 2
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 2048
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_fastopen_key = 998648db-d6ae5b92-63b255b0-1da33fdc
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_limit_output_bytes = 262144
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 4096
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_max_tw_buckets = 10000
net.ipv4.tcp_mem = 11838        15784   23676
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096        87380   134217728
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096        65536   134217728
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_mem = 23676        31568   47352
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.xfrm4_gc_thresh = 2147483647
net.ipv6.anycast_src_echo_reply = 0
net.ipv6.auto_flowlabels = 1
net.ipv6.bindv6only = 0
net.ipv6.conf.all.accept_dad = 1
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_from_local = 0
net.ipv6.conf.all.accept_ra_min_hop_limit = 1
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.all.drop_unsolicited_na = 0
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.ignore_routes_with_linkdown = 0
net.ipv6.conf.all.keep_addr_on_down = 0
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitation_max_interval = 3600
net.ipv6.conf.all.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.all.stable_secret': I/O error
net.ipv6.conf.all.suppress_frag_ndisc = 1
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_oif_addrs_only = 0
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_from_local = 0
net.ipv6.conf.default.accept_ra_min_hop_limit = 1
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.default.drop_unsolicited_na = 0
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.ignore_routes_with_linkdown = 0
net.ipv6.conf.default.keep_addr_on_down = 0
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitation_max_interval = 3600
net.ipv6.conf.default.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.default.stable_secret': I/O error
net.ipv6.conf.default.suppress_frag_ndisc = 1
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_oif_addrs_only = 0
net.ipv6.conf.default.use_tempaddr = 0
net.ipv6.conf.eth0.accept_dad = 1
net.ipv6.conf.eth0.accept_ra = 1
net.ipv6.conf.eth0.accept_ra_defrtr = 1
net.ipv6.conf.eth0.accept_ra_from_local = 0
net.ipv6.conf.eth0.accept_ra_min_hop_limit = 1
net.ipv6.conf.eth0.accept_ra_mtu = 1
net.ipv6.conf.eth0.accept_ra_pinfo = 1
net.ipv6.conf.eth0.accept_redirects = 1
net.ipv6.conf.eth0.accept_source_route = 0
net.ipv6.conf.eth0.autoconf = 1
net.ipv6.conf.eth0.dad_transmits = 1
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.eth0.drop_unsolicited_na = 0
net.ipv6.conf.eth0.force_mld_version = 0
net.ipv6.conf.eth0.force_tllao = 0
net.ipv6.conf.eth0.forwarding = 1
net.ipv6.conf.eth0.hop_limit = 64
net.ipv6.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv6.conf.eth0.keep_addr_on_down = 0
net.ipv6.conf.eth0.max_addresses = 16
net.ipv6.conf.eth0.max_desync_factor = 600
net.ipv6.conf.eth0.mc_forwarding = 0
net.ipv6.conf.eth0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.eth0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.eth0.mtu = 1500
net.ipv6.conf.eth0.ndisc_notify = 0
net.ipv6.conf.eth0.proxy_ndp = 0
net.ipv6.conf.eth0.regen_max_retry = 3
net.ipv6.conf.eth0.router_solicitation_delay = 1
net.ipv6.conf.eth0.router_solicitation_interval = 4
net.ipv6.conf.eth0.router_solicitation_max_interval = 3600
net.ipv6.conf.eth0.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.eth0.stable_secret': I/O error
net.ipv6.conf.eth0.suppress_frag_ndisc = 1
net.ipv6.conf.eth0.temp_prefered_lft = 86400
net.ipv6.conf.eth0.temp_valid_lft = 604800
net.ipv6.conf.eth0.use_oif_addrs_only = 0
net.ipv6.conf.eth0.use_tempaddr = 0
net.ipv6.conf.if0.accept_dad = 1
net.ipv6.conf.if0.accept_ra = 1
net.ipv6.conf.if0.accept_ra_defrtr = 1
net.ipv6.conf.if0.accept_ra_from_local = 0
net.ipv6.conf.if0.accept_ra_min_hop_limit = 1
net.ipv6.conf.if0.accept_ra_mtu = 1
net.ipv6.conf.if0.accept_ra_pinfo = 1
net.ipv6.conf.if0.accept_redirects = 1
net.ipv6.conf.if0.accept_source_route = 0
net.ipv6.conf.if0.autoconf = 1
net.ipv6.conf.if0.dad_transmits = 1
net.ipv6.conf.if0.disable_ipv6 = 1
net.ipv6.conf.if0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.if0.drop_unsolicited_na = 0
net.ipv6.conf.if0.force_mld_version = 0
net.ipv6.conf.if0.force_tllao = 0
net.ipv6.conf.if0.forwarding = 1
net.ipv6.conf.if0.hop_limit = 64
net.ipv6.conf.if0.ignore_routes_with_linkdown = 0
net.ipv6.conf.if0.keep_addr_on_down = 0
net.ipv6.conf.if0.max_addresses = 16
net.ipv6.conf.if0.max_desync_factor = 600
net.ipv6.conf.if0.mc_forwarding = 0
net.ipv6.conf.if0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.if0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.if0.mtu = 1500
net.ipv6.conf.if0.ndisc_notify = 0
net.ipv6.conf.if0.proxy_ndp = 0
net.ipv6.conf.if0.regen_max_retry = 3
net.ipv6.conf.if0.router_solicitation_delay = 1
net.ipv6.conf.if0.router_solicitation_interval = 4
net.ipv6.conf.if0.router_solicitation_max_interval = 3600
net.ipv6.conf.if0.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.if0.stable_secret': I/O error
net.ipv6.conf.if0.suppress_frag_ndisc = 1
net.ipv6.conf.if0.temp_prefered_lft = 86400
net.ipv6.conf.if0.temp_valid_lft = 604800
net.ipv6.conf.if0.use_oif_addrs_only = 0
net.ipv6.conf.if0.use_tempaddr = 0
net.ipv6.conf.if1.accept_dad = 1
net.ipv6.conf.if1.accept_ra = 1
net.ipv6.conf.if1.accept_ra_defrtr = 1
net.ipv6.conf.if1.accept_ra_from_local = 0
net.ipv6.conf.if1.accept_ra_min_hop_limit = 1
net.ipv6.conf.if1.accept_ra_mtu = 1
net.ipv6.conf.if1.accept_ra_pinfo = 1
net.ipv6.conf.if1.accept_redirects = 1
net.ipv6.conf.if1.accept_source_route = 0
net.ipv6.conf.if1.autoconf = 1
net.ipv6.conf.if1.dad_transmits = 1
net.ipv6.conf.if1.disable_ipv6 = 1
net.ipv6.conf.if1.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.if1.drop_unsolicited_na = 0
net.ipv6.conf.if1.force_mld_version = 0
net.ipv6.conf.if1.force_tllao = 0
net.ipv6.conf.if1.forwarding = 1
net.ipv6.conf.if1.hop_limit = 64
net.ipv6.conf.if1.ignore_routes_with_linkdown = 0
net.ipv6.conf.if1.keep_addr_on_down = 0
net.ipv6.conf.if1.max_addresses = 16
net.ipv6.conf.if1.max_desync_factor = 600
net.ipv6.conf.if1.mc_forwarding = 0
net.ipv6.conf.if1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.if1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.if1.mtu = 1500
net.ipv6.conf.if1.ndisc_notify = 0
net.ipv6.conf.if1.proxy_ndp = 0
net.ipv6.conf.if1.regen_max_retry = 3
net.ipv6.conf.if1.router_solicitation_delay = 1
net.ipv6.conf.if1.router_solicitation_interval = 4
net.ipv6.conf.if1.router_solicitation_max_interval = 3600
net.ipv6.conf.if1.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.if1.stable_secret': I/O error
net.ipv6.conf.if1.suppress_frag_ndisc = 1
net.ipv6.conf.if1.temp_prefered_lft = 86400
net.ipv6.conf.if1.temp_valid_lft = 604800
net.ipv6.conf.if1.use_oif_addrs_only = 0
net.ipv6.conf.if1.use_tempaddr = 0
net.ipv6.conf.ifb0.accept_dad = -1
net.ipv6.conf.ifb0.accept_ra = 1
net.ipv6.conf.ifb0.accept_ra_defrtr = 1
net.ipv6.conf.ifb0.accept_ra_from_local = 0
net.ipv6.conf.ifb0.accept_ra_min_hop_limit = 1
net.ipv6.conf.ifb0.accept_ra_mtu = 1
net.ipv6.conf.ifb0.accept_ra_pinfo = 1
net.ipv6.conf.ifb0.accept_redirects = 1
net.ipv6.conf.ifb0.accept_source_route = 0
net.ipv6.conf.ifb0.autoconf = 1
net.ipv6.conf.ifb0.dad_transmits = 1
net.ipv6.conf.ifb0.disable_ipv6 = 1
net.ipv6.conf.ifb0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.ifb0.drop_unsolicited_na = 0
net.ipv6.conf.ifb0.force_mld_version = 0
net.ipv6.conf.ifb0.force_tllao = 0
net.ipv6.conf.ifb0.forwarding = 1
net.ipv6.conf.ifb0.hop_limit = 64
net.ipv6.conf.ifb0.ignore_routes_with_linkdown = 0
net.ipv6.conf.ifb0.keep_addr_on_down = 0
net.ipv6.conf.ifb0.max_addresses = 16
net.ipv6.conf.ifb0.max_desync_factor = 600
net.ipv6.conf.ifb0.mc_forwarding = 0
net.ipv6.conf.ifb0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.ifb0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.ifb0.mtu = 1500
net.ipv6.conf.ifb0.ndisc_notify = 0
net.ipv6.conf.ifb0.proxy_ndp = 0
net.ipv6.conf.ifb0.regen_max_retry = 3
net.ipv6.conf.ifb0.router_solicitation_delay = 1
net.ipv6.conf.ifb0.router_solicitation_interval = 4
net.ipv6.conf.ifb0.router_solicitation_max_interval = 3600
net.ipv6.conf.ifb0.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.ifb0.stable_secret': I/O error
net.ipv6.conf.ifb0.suppress_frag_ndisc = 1
net.ipv6.conf.ifb0.temp_prefered_lft = 86400
net.ipv6.conf.ifb0.temp_valid_lft = 604800
net.ipv6.conf.ifb0.use_oif_addrs_only = 0
net.ipv6.conf.ifb0.use_tempaddr = 0
net.ipv6.conf.ifb1.accept_dad = -1
net.ipv6.conf.ifb1.accept_ra = 1
net.ipv6.conf.ifb1.accept_ra_defrtr = 1
net.ipv6.conf.ifb1.accept_ra_from_local = 0
net.ipv6.conf.ifb1.accept_ra_min_hop_limit = 1
net.ipv6.conf.ifb1.accept_ra_mtu = 1
net.ipv6.conf.ifb1.accept_ra_pinfo = 1
net.ipv6.conf.ifb1.accept_redirects = 1
net.ipv6.conf.ifb1.accept_source_route = 0
net.ipv6.conf.ifb1.autoconf = 1
net.ipv6.conf.ifb1.dad_transmits = 1
net.ipv6.conf.ifb1.disable_ipv6 = 1
net.ipv6.conf.ifb1.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.ifb1.drop_unsolicited_na = 0
net.ipv6.conf.ifb1.force_mld_version = 0
net.ipv6.conf.ifb1.force_tllao = 0
net.ipv6.conf.ifb1.forwarding = 1
net.ipv6.conf.ifb1.hop_limit = 64
net.ipv6.conf.ifb1.ignore_routes_with_linkdown = 0
net.ipv6.conf.ifb1.keep_addr_on_down = 0
net.ipv6.conf.ifb1.max_addresses = 16
net.ipv6.conf.ifb1.max_desync_factor = 600
net.ipv6.conf.ifb1.mc_forwarding = 0
net.ipv6.conf.ifb1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.ifb1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.ifb1.mtu = 1500
net.ipv6.conf.ifb1.ndisc_notify = 0
net.ipv6.conf.ifb1.proxy_ndp = 0
net.ipv6.conf.ifb1.regen_max_retry = 3
net.ipv6.conf.ifb1.router_solicitation_delay = 1
net.ipv6.conf.ifb1.router_solicitation_interval = 4
net.ipv6.conf.ifb1.router_solicitation_max_interval = 3600
net.ipv6.conf.ifb1.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.ifb1.stable_secret': I/O error
net.ipv6.conf.ifb1.suppress_frag_ndisc = 1
net.ipv6.conf.ifb1.temp_prefered_lft = 86400
net.ipv6.conf.ifb1.temp_valid_lft = 604800
net.ipv6.conf.ifb1.use_oif_addrs_only = 0
net.ipv6.conf.ifb1.use_tempaddr = 0
net.ipv6.conf.lan.accept_dad = 1
net.ipv6.conf.lan.accept_ra = 1
net.ipv6.conf.lan.accept_ra_defrtr = 1
net.ipv6.conf.lan.accept_ra_from_local = 0
net.ipv6.conf.lan.accept_ra_min_hop_limit = 1
net.ipv6.conf.lan.accept_ra_mtu = 1
net.ipv6.conf.lan.accept_ra_pinfo = 1
net.ipv6.conf.lan.accept_redirects = 1
net.ipv6.conf.lan.accept_source_route = 0
net.ipv6.conf.lan.autoconf = 1
net.ipv6.conf.lan.dad_transmits = 1
net.ipv6.conf.lan.disable_ipv6 = 1
net.ipv6.conf.lan.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.lan.drop_unsolicited_na = 0
net.ipv6.conf.lan.force_mld_version = 0
net.ipv6.conf.lan.force_tllao = 0
net.ipv6.conf.lan.forwarding = 1
net.ipv6.conf.lan.hop_limit = 64
net.ipv6.conf.lan.ignore_routes_with_linkdown = 0
net.ipv6.conf.lan.keep_addr_on_down = 0
net.ipv6.conf.lan.max_addresses = 16
net.ipv6.conf.lan.max_desync_factor = 600
net.ipv6.conf.lan.mc_forwarding = 0
net.ipv6.conf.lan.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lan.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lan.mtu = 1500
net.ipv6.conf.lan.ndisc_notify = 0
net.ipv6.conf.lan.proxy_ndp = 0
net.ipv6.conf.lan.regen_max_retry = 3
net.ipv6.conf.lan.router_solicitation_delay = 1
net.ipv6.conf.lan.router_solicitation_interval = 4
net.ipv6.conf.lan.router_solicitation_max_interval = 3600
net.ipv6.conf.lan.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.lan.stable_secret': I/O error
net.ipv6.conf.lan.suppress_frag_ndisc = 1
net.ipv6.conf.lan.temp_prefered_lft = 86400
net.ipv6.conf.lan.temp_valid_lft = 604800
net.ipv6.conf.lan.use_oif_addrs_only = 0
net.ipv6.conf.lan.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_from_local = 0
net.ipv6.conf.lo.accept_ra_min_hop_limit = 1
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.lo.drop_unsolicited_na = 0
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 1
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
net.ipv6.conf.lo.keep_addr_on_down = 0
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitation_max_interval = 3600
net.ipv6.conf.lo.router_solicitations = -1
sysctl: error reading key 'net.ipv6.conf.lo.stable_secret': I/O error
net.ipv6.conf.lo.suppress_frag_ndisc = 1
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_oif_addrs_only = 0
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.conf.tun0.accept_dad = -1
net.ipv6.conf.tun0.accept_ra = 1
net.ipv6.conf.tun0.accept_ra_defrtr = 1
net.ipv6.conf.tun0.accept_ra_from_local = 0
net.ipv6.conf.tun0.accept_ra_min_hop_limit = 1
net.ipv6.conf.tun0.accept_ra_mtu = 1
net.ipv6.conf.tun0.accept_ra_pinfo = 1
net.ipv6.conf.tun0.accept_redirects = 1
net.ipv6.conf.tun0.accept_source_route = 0
net.ipv6.conf.tun0.autoconf = 1
net.ipv6.conf.tun0.dad_transmits = 1
net.ipv6.conf.tun0.disable_ipv6 = 0
net.ipv6.conf.tun0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.tun0.drop_unsolicited_na = 0
net.ipv6.conf.tun0.force_mld_version = 0
net.ipv6.conf.tun0.force_tllao = 0
net.ipv6.conf.tun0.forwarding = 1
net.ipv6.conf.tun0.hop_limit = 64
net.ipv6.conf.tun0.ignore_routes_with_linkdown = 0
net.ipv6.conf.tun0.keep_addr_on_down = 0
net.ipv6.conf.tun0.max_addresses = 16
net.ipv6.conf.tun0.max_desync_factor = 600
net.ipv6.conf.tun0.mc_forwarding = 0
net.ipv6.conf.tun0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.tun0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.tun0.mtu = 1450
net.ipv6.conf.tun0.ndisc_notify = 0
net.ipv6.conf.tun0.proxy_ndp = 0
net.ipv6.conf.tun0.regen_max_retry = 3
net.ipv6.conf.tun0.router_solicitation_delay = 1
net.ipv6.conf.tun0.router_solicitation_interval = 4
net.ipv6.conf.tun0.router_solicitation_max_interval = 3600
net.ipv6.conf.tun0.router_solicitations = -1
net.ipv6.conf.tun0.stable_secret = 34cc:f6dd:830a:b5b8:ca95:6e02:ebea:7d9d
net.ipv6.conf.tun0.suppress_frag_ndisc = 1
net.ipv6.conf.tun0.temp_prefered_lft = 86400
net.ipv6.conf.tun0.temp_valid_lft = 604800
net.ipv6.conf.tun0.use_oif_addrs_only = 0
net.ipv6.conf.tun0.use_tempaddr = -1
net.ipv6.conf.xtun0.accept_dad = -1
net.ipv6.conf.xtun0.accept_ra = 1
net.ipv6.conf.xtun0.accept_ra_defrtr = 1
net.ipv6.conf.xtun0.accept_ra_from_local = 0
net.ipv6.conf.xtun0.accept_ra_min_hop_limit = 1
net.ipv6.conf.xtun0.accept_ra_mtu = 1
net.ipv6.conf.xtun0.accept_ra_pinfo = 1
net.ipv6.conf.xtun0.accept_redirects = 1
net.ipv6.conf.xtun0.accept_source_route = 0
net.ipv6.conf.xtun0.autoconf = 1
net.ipv6.conf.xtun0.dad_transmits = 1
net.ipv6.conf.xtun0.disable_ipv6 = 0
net.ipv6.conf.xtun0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.xtun0.drop_unsolicited_na = 0
net.ipv6.conf.xtun0.force_mld_version = 0
net.ipv6.conf.xtun0.force_tllao = 0
net.ipv6.conf.xtun0.forwarding = 1
net.ipv6.conf.xtun0.hop_limit = 64
net.ipv6.conf.xtun0.ignore_routes_with_linkdown = 0
net.ipv6.conf.xtun0.keep_addr_on_down = 0
net.ipv6.conf.xtun0.max_addresses = 16
net.ipv6.conf.xtun0.max_desync_factor = 600
net.ipv6.conf.xtun0.mc_forwarding = 0
net.ipv6.conf.xtun0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.xtun0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.xtun0.mtu = 1400
net.ipv6.conf.xtun0.ndisc_notify = 0
net.ipv6.conf.xtun0.proxy_ndp = 0
net.ipv6.conf.xtun0.regen_max_retry = 3
net.ipv6.conf.xtun0.router_solicitation_delay = 1
net.ipv6.conf.xtun0.router_solicitation_interval = 4
net.ipv6.conf.xtun0.router_solicitation_max_interval = 3600
net.ipv6.conf.xtun0.router_solicitations = -1
net.ipv6.conf.xtun0.stable_secret = f033:6250:3eba:4688:4d61:db43:e4c9:6a2a
net.ipv6.conf.xtun0.suppress_frag_ndisc = 1
net.ipv6.conf.xtun0.temp_prefered_lft = 86400
net.ipv6.conf.xtun0.temp_valid_lft = 604800
net.ipv6.conf.xtun0.use_oif_addrs_only = 0
net.ipv6.conf.xtun0.use_tempaddr = -1
net.ipv6.flowlabel_consistency = 1
net.ipv6.flowlabel_state_ranges = 0
net.ipv6.fwmark_reflect = 0
net.ipv6.icmp.ratelimit = 1000
net.ipv6.idgen_delay = 1
net.ipv6.idgen_retries = 3
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 0
net.ipv6.ip6frag_time = 60
net.ipv6.ip_nonlocal_bind = 0
net.ipv6.mld_max_msf = 64
net.ipv6.mld_qrv = 2
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time = 30
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_resolicit = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time = 100
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 30
net.ipv6.neigh.default.unres_qlen_bytes = 65536
net.ipv6.neigh.eth0.anycast_delay = 100
net.ipv6.neigh.eth0.app_solicit = 0
net.ipv6.neigh.eth0.base_reachable_time = 30
net.ipv6.neigh.eth0.base_reachable_time_ms = 30000
net.ipv6.neigh.eth0.delay_first_probe_time = 5
net.ipv6.neigh.eth0.gc_stale_time = 60
net.ipv6.neigh.eth0.locktime = 0
net.ipv6.neigh.eth0.mcast_resolicit = 0
net.ipv6.neigh.eth0.mcast_solicit = 3
net.ipv6.neigh.eth0.proxy_delay = 80
net.ipv6.neigh.eth0.proxy_qlen = 64
net.ipv6.neigh.eth0.retrans_time = 100
net.ipv6.neigh.eth0.retrans_time_ms = 1000
net.ipv6.neigh.eth0.ucast_solicit = 3
net.ipv6.neigh.eth0.unres_qlen = 30
net.ipv6.neigh.eth0.unres_qlen_bytes = 65536
net.ipv6.neigh.if0.anycast_delay = 100
net.ipv6.neigh.if0.app_solicit = 0
net.ipv6.neigh.if0.base_reachable_time = 30
net.ipv6.neigh.if0.base_reachable_time_ms = 30000
net.ipv6.neigh.if0.delay_first_probe_time = 5
net.ipv6.neigh.if0.gc_stale_time = 60
net.ipv6.neigh.if0.locktime = 0
net.ipv6.neigh.if0.mcast_resolicit = 0
net.ipv6.neigh.if0.mcast_solicit = 3
net.ipv6.neigh.if0.proxy_delay = 80
net.ipv6.neigh.if0.proxy_qlen = 64
net.ipv6.neigh.if0.retrans_time = 100
net.ipv6.neigh.if0.retrans_time_ms = 1000
net.ipv6.neigh.if0.ucast_solicit = 3
net.ipv6.neigh.if0.unres_qlen = 30
net.ipv6.neigh.if0.unres_qlen_bytes = 65536
net.ipv6.neigh.if1.anycast_delay = 100
net.ipv6.neigh.if1.app_solicit = 0
net.ipv6.neigh.if1.base_reachable_time = 30
net.ipv6.neigh.if1.base_reachable_time_ms = 30000
net.ipv6.neigh.if1.delay_first_probe_time = 5
net.ipv6.neigh.if1.gc_stale_time = 60
net.ipv6.neigh.if1.locktime = 0
net.ipv6.neigh.if1.mcast_resolicit = 0
net.ipv6.neigh.if1.mcast_solicit = 3
net.ipv6.neigh.if1.proxy_delay = 80
net.ipv6.neigh.if1.proxy_qlen = 64
net.ipv6.neigh.if1.retrans_time = 100
net.ipv6.neigh.if1.retrans_time_ms = 1000
net.ipv6.neigh.if1.ucast_solicit = 3
net.ipv6.neigh.if1.unres_qlen = 30
net.ipv6.neigh.if1.unres_qlen_bytes = 65536
net.ipv6.neigh.ifb0.anycast_delay = 100
net.ipv6.neigh.ifb0.app_solicit = 0
net.ipv6.neigh.ifb0.base_reachable_time = 30
net.ipv6.neigh.ifb0.base_reachable_time_ms = 30000
net.ipv6.neigh.ifb0.delay_first_probe_time = 5
net.ipv6.neigh.ifb0.gc_stale_time = 60
net.ipv6.neigh.ifb0.locktime = 0
net.ipv6.neigh.ifb0.mcast_resolicit = 0
net.ipv6.neigh.ifb0.mcast_solicit = 3
net.ipv6.neigh.ifb0.proxy_delay = 80
net.ipv6.neigh.ifb0.proxy_qlen = 64
net.ipv6.neigh.ifb0.retrans_time = 100
net.ipv6.neigh.ifb0.retrans_time_ms = 1000
net.ipv6.neigh.ifb0.ucast_solicit = 3
net.ipv6.neigh.ifb0.unres_qlen = 30
net.ipv6.neigh.ifb0.unres_qlen_bytes = 65536
net.ipv6.neigh.ifb1.anycast_delay = 100
net.ipv6.neigh.ifb1.app_solicit = 0
net.ipv6.neigh.ifb1.base_reachable_time = 30
net.ipv6.neigh.ifb1.base_reachable_time_ms = 30000
net.ipv6.neigh.ifb1.delay_first_probe_time = 5
net.ipv6.neigh.ifb1.gc_stale_time = 60
net.ipv6.neigh.ifb1.locktime = 0
net.ipv6.neigh.ifb1.mcast_resolicit = 0
net.ipv6.neigh.ifb1.mcast_solicit = 3
net.ipv6.neigh.ifb1.proxy_delay = 80
net.ipv6.neigh.ifb1.proxy_qlen = 64
net.ipv6.neigh.ifb1.retrans_time = 100
net.ipv6.neigh.ifb1.retrans_time_ms = 1000
net.ipv6.neigh.ifb1.ucast_solicit = 3
net.ipv6.neigh.ifb1.unres_qlen = 30
net.ipv6.neigh.ifb1.unres_qlen_bytes = 65536
net.ipv6.neigh.lan.anycast_delay = 100
net.ipv6.neigh.lan.app_solicit = 0
net.ipv6.neigh.lan.base_reachable_time = 30
net.ipv6.neigh.lan.base_reachable_time_ms = 30000
net.ipv6.neigh.lan.delay_first_probe_time = 5
net.ipv6.neigh.lan.gc_stale_time = 60
net.ipv6.neigh.lan.locktime = 0
net.ipv6.neigh.lan.mcast_resolicit = 0
net.ipv6.neigh.lan.mcast_solicit = 3
net.ipv6.neigh.lan.proxy_delay = 80
net.ipv6.neigh.lan.proxy_qlen = 64
net.ipv6.neigh.lan.retrans_time = 100
net.ipv6.neigh.lan.retrans_time_ms = 1000
net.ipv6.neigh.lan.ucast_solicit = 3
net.ipv6.neigh.lan.unres_qlen = 30
net.ipv6.neigh.lan.unres_qlen_bytes = 65536
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time = 30
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_resolicit = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time = 100
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 30
net.ipv6.neigh.lo.unres_qlen_bytes = 65536
net.ipv6.neigh.tun0.anycast_delay = 100
net.ipv6.neigh.tun0.app_solicit = 0
net.ipv6.neigh.tun0.base_reachable_time = 30
net.ipv6.neigh.tun0.base_reachable_time_ms = 30000
net.ipv6.neigh.tun0.delay_first_probe_time = 5
net.ipv6.neigh.tun0.gc_stale_time = 60
net.ipv6.neigh.tun0.locktime = 0
net.ipv6.neigh.tun0.mcast_resolicit = 0
net.ipv6.neigh.tun0.mcast_solicit = 3
net.ipv6.neigh.tun0.proxy_delay = 80
net.ipv6.neigh.tun0.proxy_qlen = 64
net.ipv6.neigh.tun0.retrans_time = 100
net.ipv6.neigh.tun0.retrans_time_ms = 1000
net.ipv6.neigh.tun0.ucast_solicit = 3
net.ipv6.neigh.tun0.unres_qlen = 30
net.ipv6.neigh.tun0.unres_qlen_bytes = 65536
net.ipv6.neigh.xtun0.anycast_delay = 100
net.ipv6.neigh.xtun0.app_solicit = 0
net.ipv6.neigh.xtun0.base_reachable_time = 30
net.ipv6.neigh.xtun0.base_reachable_time_ms = 30000
net.ipv6.neigh.xtun0.delay_first_probe_time = 5
net.ipv6.neigh.xtun0.gc_stale_time = 60
net.ipv6.neigh.xtun0.locktime = 0
net.ipv6.neigh.xtun0.mcast_resolicit = 0
net.ipv6.neigh.xtun0.mcast_solicit = 3
net.ipv6.neigh.xtun0.proxy_delay = 80
net.ipv6.neigh.xtun0.proxy_qlen = 64
net.ipv6.neigh.xtun0.retrans_time = 100
net.ipv6.neigh.xtun0.retrans_time_ms = 1000
net.ipv6.neigh.xtun0.ucast_solicit = 3
net.ipv6.neigh.xtun0.unres_qlen = 30
net.ipv6.neigh.xtun0.unres_qlen_bytes = 65536
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.xfrm6_gc_thresh = 2147483647
net.mptcp.mptcp_binder_gateways =
net.mptcp.mptcp_checksum = 1
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = default
net.mptcp.mptcp_syn_retries = 3
net.mptcp.mptcp_version = 0
net.netfilter.nf_conntrack_acct = 1
net.netfilter.nf_conntrack_buckets = 8192
net.netfilter.nf_conntrack_checksum = 0
net.netfilter.nf_conntrack_count = 518
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 128
net.netfilter.nf_conntrack_frag6_high_thresh = 4194304
net.netfilter.nf_conntrack_frag6_low_thresh = 3145728
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_no_window_check = 1
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 7440
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = nf_log_ipv6
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = nf_log_ipv4
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.nf_conntrack_max = 131072
net.sctp.addip_enable = 0
net.sctp.addip_noauth_enable = 0
net.sctp.addr_scope_policy = 1
net.sctp.association_max_retrans = 10
net.sctp.auth_enable = 0
net.sctp.cookie_hmac_alg = md5
net.sctp.cookie_preserve_enable = 1
net.sctp.default_auto_asconf = 0
net.sctp.hb_interval = 30000
net.sctp.max_autoclose = 21474836
net.sctp.max_burst = 4
net.sctp.max_init_retransmits = 8
net.sctp.path_max_retrans = 5
net.sctp.pf_enable = 1
net.sctp.pf_retrans = 0
net.sctp.prsctp_enable = 1
net.sctp.rcvbuf_policy = 0
net.sctp.rto_alpha_exp_divisor = 3
net.sctp.rto_beta_exp_divisor = 2
net.sctp.rto_initial = 3000
net.sctp.rto_max = 60000
net.sctp.rto_min = 1000
net.sctp.rwnd_update_shift = 4
net.sctp.sack_timeout = 200
net.sctp.sctp_mem = 23727       31639   47454
net.sctp.sctp_rmem = 4096       961500  1012448
net.sctp.sctp_wmem = 4096       16384   1012448
net.sctp.sndbuf_policy = 0
net.sctp.valid_cookie_life = 60000
net.unix.max_dgram_qlen = 10
user.max_cgroup_namespaces = 3845
user.max_ipc_namespaces = 3845
user.max_mnt_namespaces = 3845
user.max_net_namespaces = 3845
user.max_pid_namespaces = 3845
user.max_user_namespaces = 3845
user.max_uts_namespaces = 3845
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256   256     32
vm.max_map_count = 65530
vm.min_free_kbytes = 16384
vm.mmap_min_addr = 4096
vm.mmap_rnd_bits = 28
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 31561
vm.vfs_cache_pressure = 100
vm.watermark_scale_factor = 10
Ysurac commented 6 years ago

Thanks. The only major diff I can see is the use of cubic for TCP congestion instead of olia. You can try that in Network->MPTCP. Sometimes cubic is better.

fnoir commented 6 years ago

Same result:

curl http://ovh.net/files/1Gio.dat -o nul
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  4 1024M    4 49.3M    0     0  1626k      0  0:10:44  0:00:31  0:10:13 1577k

speedtest: down => 50 Mbps i download a game with steam => 72 Mbps Oo

Ysurac commented 6 years ago

Steam use multiples connections, it's why this use the full available bandwidth. You can try pre-release: https://www.openmptcprouter.com/release/targets/ I was suggesting cubic for @SoGooDFR because he doesn't use aggregation.

fnoir commented 6 years ago

I install v0.27-1-g06324c5 same result:

curl http://ovh.net/files/10Gio.dat -o nul
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2 10.0G    2  263M    0     0  1504k      0  1:56:11  0:02:59  1:53:12 1484k
SoGooDFR commented 6 years ago 
curl http://ovh.net/files/1Gio.dat -o nul
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  7 1024M    7 75.9M    0     0  2680k      0  0:06:31  0:00:29  0:06:02 2714k^C

I've already test cubic from the past, she is unstable with Glorytun, i had ping spike issue. This is why i leave OTB. I view a small improvement with openmptcprouter v0.27-1-g06324c5 r7308-ebe1216c7c, but the problem persist, maybe a VPS issue to ?

Interesting, i've download the lastest Cuisine Royale update from game launcher and it will give to me 9,2 mo/s. But on direct download (from curl http://ovh.net/files/10Gio.dat -o nul) i'm stuck at 2,6-2,7 mo/s.

Ysurac commented 6 years ago

I have the same problem when using the modem directly (without using OpenMPTCProuter, but still on the network). Sometimes it's better but after 2-3 tests this is also about 1500k/s. Maybe a limit after more than x connections are opened at the same time with 4G ?

I will make more tests (if this is really the problem ? how many connections ? UDP and TCP or only TCP ? ...), and check what I can do about that.

SoGooDFR commented 6 years ago

In my case, the limit is the antenna cell 😄 Actually i use direct modem connection with OPNSense firewall, i've no problem from october 2017.

I test udp speed with a torrent on OpenMPTCProuter - Glorytun UDP : openmptcp_udp

Ysurac commented 6 years ago

By default Glorytun TCP is used for UDP, so this also use MPTCP. 9.2Mio/s (77Mb/s) seems quite good. But like Steam, this open many connections at the same time.

Direct modem connection maybe open less connection than what I open with OpenMPTCProuter with ShadowSocks by default.

With Steam I often get about 85-90Mb/s too (modem limit to 100Mb/s), but direct download is far slower... I hope to find why...

SoGooDFR commented 6 years ago

For me UDP from TCP is the hell and generate easily ping spikes. I've already test shadowsocks only on debian without OpenMPTCP installed and i get my full connection.

Ysurac commented 6 years ago

You can set Glorytun UDP in the wizard, this will not use TCP/MPTCP at all.