Chadster766 commented 4 years ago

McDebian 5.6.14 Beta

Updates:

Linux Kernel 5.6.14
Implementing nftables flowtable offload which increases the DSA switch wan to lan routing to gigabit speeds for single connections (didn't work in previous kernel versions I tried)

Notes:

I recommend that only users that have TTL access to their WRT routers do McDebian beta testing.

In the WRT1900AC V1 make sure you have the below u-boot envars set to to accommodate the increased kernel size.

#This is what I have my WRT1900AC V1 u-boot kernel size set for
root@MCDEBIAN:~# fw_printenv pri_kern_size
pri_kern_size=0x520000
root@MCDEBIAN:~# fw_printenv alt_kern_size
alt_kern_size=0x520000

Fixes kernel "oops in br_vlan_enabled" Update Testing has shown that the below point doesn't work for the WRT32000ACM V1 but works on the WRT3200ACM V2.
Kernel version brings up interfaces properly so "pre-up ifup --ignore-errors br0" is no longer required in wan config and br0 config can be set to "auto br0".

Edit file "/etc/nftables.conf" to implement nftables flowtable offload

create table inet x
add flowtable inet x f { hook ingress priority 0; devices = { wan, br0 }; }
add chain inet x y { type filter hook forward priority 0; policy accept; }
add rule inet x y ip protocol { udp, tcp } flow offload @f
add rule inet x y counter packets 0 bytes 0

Add "post-up nft -f /etc/nftables.conf" to br0 config to implement nftables flowtable offload (don't enable nftables.service since it fails due to starting before all DSA interfaces are up"

Firmware:

wget --user=mcdebian --password=mcdebian123 http://www.protechs-online.com/downloads/McDebian/firmwares/McDebian-Buster-WRT1900AC-V1-FW_VER1_kernel_5.6.14.img

wget --user=mcdebian --password=mcdebian123 http://www.protechs-online.com/downloads/McDebian/firmwares/McDebian-Buster-WRT1900AC-V2-FW_VER1_kernel_5.6.14.img

wget --user=mcdebian --password=mcdebian123 http://www.protechs-online.com/downloads/McDebian/firmwares/McDebian-Buster-WRT1200AC-V1-FW_VER1_kernel_5.6.14.img

wget --user=mcdebian --password=mcdebian123 http://www.protechs-online.com/downloads/McDebian/firmwares/McDebian-Buster-WRT3200ACM-V1-FW_VER1_kernel_5.6.14.img

wget --user=mcdebian --password=mcdebian123 http://www.protechs-online.com/downloads/McDebian/firmwares/McDebian-Buster-WRT32X-V1-FW_VER1_kernel_5.6.14.img

Root File System Update:

wget --user=mcdebian --password=mcdebian123 http://www.protechs-online.com/downloads/McDebian/rootfs-updates/mcdebian-rootfs-kernel-modules-firmwares-updates-v5.6.14-ver1.gz

rm -rf /lib/modules/*
rm -rf /lib/firmware

tar zxfk mcdebian-rootfs-kernel-modules-firmwares-updates-v5.6.14-ver1.gz

IPv6

To enable IPv6 in this beta release you need to enable radvd:

systemctl enable radvd

Then uncomment the IPv6 config lines in:

vim /etc/default/isc-dhcp-server

After that reboot the router.

Chadster766 commented 4 years ago

Thank you very much for your help.

Nothing more needed.

Mark-GR73 commented 4 years ago

Thank you too. First for your work , Second for the results . and last for the nice team work... We 'll keep in touch.

Mark-GR73 commented 4 years ago

just look for inform systemctl status systemd-networkd-wait-online if to enable it

maybe resolve the problem to nftables and do not need to delay the post-up on interface

sfrost commented 4 years ago

Increasing the kern size did indeed work for me, thanks so much!

Chadster766 commented 4 years ago

Thanks @sfrost

Chadster766 commented 4 years ago

And @ValCher1961

Chadster766 commented 4 years ago

Oops

sfrost commented 4 years ago

I've also been able to get this up and running on a WRT3200ACM, though I had some challenges- loading the 5.6.14 kernel from Linux ended up with a CRC error, but booting into the bootloader and tftp'ing the kernel ended up working and I was able to boot.

I have had some challenges getting it to actually boot sometimes... I think the problem is that when the serial console is connected but the pins aren't positioned exactly right, it gets hung up trying to write on the console and ends up not working. Fiddling with the cables usually gets me to working. What are you using for the connection...? If it's something more form-fitting and secure, I'd love to know where I can get it, so I can have more ready and reliable serial connections on my routers (I'm now up to 2 1900's and a 3200...).

I noticed your question about where to take the project at this point- personally, I'd really love to have it be a fully working and self-reliable system, and for that, it seems to me anyway, that we really need to be able to build the kernels ourselves properly as Debian packages and install them... Maybe even get the correct kernels formally uploaded to Debian and included in future Debian releases and maintained as part of Debian. Hopefully the kernel install scripts could even be adapted to handle updating the boot loader and such when new kernels are released, so I can just 'apt-get upgrade' like I do on most of my other Debian-based systems. I'm a DD, if that is helpful at all, lol.

sfrost commented 4 years ago

I was also able to get 5.6.14 up and running on the first try with the updated pri_kern_size/alt_kern_size on my other 1900! Looking good!

sfrost commented 4 years ago

Using bridges with VLANs and everything seems to be working great under 5.6.14 too, and I've migrated everything over to nftables and it's looking all very good, thanks so much for this!

Mark-GR73 commented 4 years ago

Finally all bugs are resolved? This version will remain on test ?

Chadster766 commented 4 years ago

I'm waiting for a new long term linux kernel to be release before making a new production release. I think that should happen soon since it's about time for that to happen.

Mark-GR73 commented 4 years ago

We expect this version. And of course would be nice the big part of config like pppoe ,ipv6 or anything... for simple home use to have a separately script on separate scriptfolder and of course always to generate the old configs like xxx.bak. I believe that the leds must working for fast and optical maintenance ,if you'd like i have suggestions or samples to enclosure in new version. Thanks for your nice work.

Chadster766 commented 4 years ago

Yes it would be good to have the Internet LED work in some way.

ValCher1961 commented 4 years ago

Hello @Chadster766 and @All! I hope you're okay. Chad, you wrote - "Yes it would be good to have the Internet LED work in some way" There is a key to the ledd controls - CONFIG_LEDS_TRIGGER_NETDEV=y and add to rc.local -

echo wan > /sys/class/leds/rango:white:wan/device_name
echo 1 > /sys/class/leds/rango:white:wan/rx
echo 1 > /sys/class/leds/rango:white:wan/tx

Or did I miss something? :)

All health.

Chadster766 commented 4 years ago

Thanks @ValCher1961 I will give it a shot 👍

Chadster766 commented 4 years ago

I'm not sure having the Internet LED flash like a normal Ethernet LED is the best.

As it is now without having CONFIG_LEDS_TRIGGER_NETDEV we can control the LED via:

/sys/class/leds/pca963x:rango:amber:wan/brightness
/sys/class/leds/pca963x:rango:white:wan/brightness

Maybe we could have a service script that if the WAN is DOWN off, UP full amber. The script could also ping a site like 8.8.8.8 periodically; success full white and failure flash fading amber.

Mark-GR73 commented 4 years ago

I Believe a script is a very good idea , we can change the time but every minute is a more precisely solution For test reason there is a folder , in this example its name is /Ascripts/cron And do not forget to erase the - in script before # and * in cronjob (For typing reasons)

-#!/bin/bash

-# sh -v /Ascripts/cron/wanled.sh -# cat /sys/class/leds/pca963x:rango:amber:wan/trigger

echo "timer" > /sys/class/leds/rango:white:power/trigger

-## Wan Led ##

echo "timer" > /sys/class/leds/pca963x:rango:amber:wan/trigger sleep 0.5 echo "timer" > /sys/class/leds/pca963x:rango:white:wan/trigger sleep 2 ping -q -c2 8.8.4.4 2>&1 >/dev/null

if [ $? -eq 0 ] then

echo "default-on" > /sys/class/leds/pca963x:rango:white:wan/trigger echo "none" > /sys/class/leds/pca963x:rango:amber:wan/trigger echo -e "\033[32m Internet, OK ! and Isp connection \033[m"

else

echo "timer" > /sys/class/leds/pca963x:rango:amber:wan/trigger sleep 0.5 echo "timer" > /sys/class/leds/pca963x:rango:white:wan/trigger echo -e "\033[31m FAIL internet connection through ISP, NO INTERNET !!! \033[m"

fi

echo "default-on" > /sys/class/leds/rango:white:power/trigger

exit 0

And of course it must be run every minute At /etc/crontab

- * root sh /Ascripts/cron/wanled.sh 2>&1 >/dev/null

Chadster766 commented 4 years ago

Thanks I will test it out

Mark-GR73 commented 4 years ago

Hi at all in the team ,it is nice to have many thoughts or opinions. Well, another idea for more pretty optical view is to make it more alive, like the SATA led in the middle of the router, to be its heart and of course to show if finnaly boot the McDebian.

-#!/bin/bash

-## sh -v /Ascripts/cron/mcdebianboot.sh -## cat /sys/class/leds/rango:white:sata/trigger

-## Heartbeat sata led echo "heartbeat" > /sys/class/leds/rango:white:sata/trigger

exit 0

It must run once on boot At /etc/crontab

@reboot root sleep 8 ; sh /Ascripts/startup/mcdebianboot.sh 2>&1 >/dev/null

And do not forget to erase the - in script before # and of course the @ in cronjob it is necessary and the delay too.

Take a look ,and any additional idea is welcome

Mark-GR73 commented 4 years ago

Take care,

In crontab the wanled.sh must be first and the mcdebianboot.sh at last

sfrost commented 3 years ago

Curious if anyone else has noticed that, after a few weeks, the wifi seems to drop using this kernel, et al? I've had 2.4ghz just stop working on the wrt3200acm, and had the 1900acm seemingly stop running entirely except for the ethernet switch. As noted in the other issue that I opened, I also haven't been able to figure out how to make roaming work for devices to be able to go between multiple routers using this build. Thoughts, other comments, other experiences, would all be most welcome and good to hear about! I'm going to try and set something up so I can catch any kernel panics and such that happen. Thanks!

Mark-GR73 commented 3 years ago

I need help with ipv6 .. I have tried alots of solutions , i try to connect with ppoe ,but nothing with ipv6. Is there something who must know about that?

Mark-GR73 commented 3 years ago

And a very useful info.. For apt ....

sudo echo '1' > /var/lib/dpkg/info/format

Good upgrades..

Chadster766 commented 3 years ago

@Mark-GR73 how do you normally connect to your ISP?

Who is your ISP so I can lookup there IPv6 support?

Mark-GR73 commented 3 years ago

Hi. I Live in Greece. I have pppoe for connectivity. My ISP is OTE or Cosmote. Be sure that there is ipv6 connectivity. I have tried alot of setups but nothing,yet..

Chadster766 commented 3 years ago

Ok not much info on OTE IPv6.

Regardless by default the WAN port in McDebian is configure to accept and forward IPv6 IP with a basic firewall config for it.

With the WRT connected to the ISP without with the McDebian network\firewall default configuration what does "ip addr sh wan" display?

Then run "pppoeconf wan" report what is found and configured. This command will automatically detect and configure the pppoe connection with some input from you.

If you have a successful pppoe connection run the "ip addr sh wan" and "ip addr sh ppp0" to get more info.

If ppp0 is connected you will need to modify your firewall rules with:

sed -i s/wan/ppp0/g /etc/iptables.up.rules
sed -i s/wan/ppp0/g /etc/ip6tables.up.rules

Once that is done you should be able to "ping -6 www.google.com". If you can at this point then it's time to configure the LAN IPv6 support with:

To enable IPv6 you need to enable radvd:

systemctl enable radvd
Then uncomment the IPv6 config lines in:

vim /etc/default/isc-dhcp-server

Mark-GR73 commented 3 years ago

Of course i follow regardly all the previous and those instructions who you adviced, and after alot of test the result until now, are....

ipv6.txt

-Something in your mind ?

Chadster766 commented 3 years ago

It looks like your ISP hasn't given you prefix 2a02:587:0:0::/64.

If they are providing DHCP-PD then your config would have to be different to figure out what IPv6 prefix the ISP is giving you.

Mark-GR73 commented 3 years ago

ISP working properly. With other routers everything is working properly. I believe your advice with DHCP-PD is exactly where i must to dig. I will give one more try...

LeBrad commented 3 years ago

Thanks for this McBeta, It's great to have nftables.

However, I'm hitting a wired speed bottleneck on my 1900AC. I'm getting gigabit link speed between computers connected to the wired LAN ports, but speed drops to around 250 megabits/second when I connect one of the computers to the WAN port and one to the LAN.

I tried this with local computers and various cables to make sure it wasn't an ISP or wire problem.

When I run an iperf3 speed test with a link between WAN and LAN the cpu usage of the 1900AC spikes to almost 100%, with lots of ksoftirqd usage.

I wonder if this is a firewall issue, because I assume that the 1900AC is fast enough to masquerade at gigabit speeds. Also it could be a kernel issue and maybe i should just wait for the next update.

Anyway, I thought I'd ask if you have any other hunches about how to remove the bottleneck.

Chadster766 commented 3 years ago

@LeBrad you will need to follow the flow offloading instruction in the below post: https://github.com/Chadster766/McDebian/issues/69#issue-628826346

To see if it's offloading the counter values will increase with internet usages on the LAN side:

nft> quit
root@MCDEBIAN:~# nft list table inet x
table inet x {
        flowtable f {
                hook ingress priority 0
                devices = { wan, br0 }
        }

        chain y {
                type filter hook forward priority 0; policy accept;
                ip protocol { tcp, udp } flow offload @f
                counter packets 1334054 bytes 177538514
        }
}

LeBrad commented 3 years ago

Thanks, hopefully my flow offloading is correct. I installed it using your instructions when I updated to the beta, and it doesn't seem to help with the bottleneck. Here's my firewall:

brad@cinefile:~$ date && sudo nft list ruleset
Mon 21 Dec 2020 05:40:09 PM UTC
table ip filter {
    chain output {
        type filter hook output priority 0; policy accept;
    }

    chain input {
        type filter hook input priority 0; policy accept;
        tcp dport { ssh, http, https } accept
        meta l4proto { tcp, udp } @th,16,16 53 accept
        iifname "br0" accept
        iifname "wan" ct state established,related accept
        iifname "wan" drop
    }

    chain forward {
        type filter hook forward priority 0; policy accept;
        tcp dport ssh accept
        tcp dport http accept
        tcp dport https accept
        tcp dport 22000 accept
        iifname "br0" accept
        iifname "wan" ct state established,related accept
        iifname "wan" drop
    }
}
table ip nat {
    chain prerouting {
        type nat hook prerouting priority 0; policy accept;
    }

    chain postrouting {
        type nat hook postrouting priority 0; policy accept;
        oifname "wan" masquerade
    }
}
table inet x {
    flowtable f {
        hook ingress priority 0
        devices = { wan, br0 }
    }

    chain y {
        type filter hook forward priority 0; policy accept;
        ip protocol { tcp, udp } flow offload @f
        counter packets 20348 bytes 2159513
    }
}

Chadster766 commented 3 years ago

@LeBrad its seem to be offloading properly.

The firewall rules look ok.

Did you add sudo to your McDebian? I wondering if you are running a ubuntu variant.

Did you test iperf3 between to laptop and the WRT? This should get gigabit no matter what.

LeBrad commented 3 years ago

Yes, I installed sudo, apt-cache says sudo: Installed: 1.8.27-1+deb10u2

Testing iperf3 directly to the McWRT was a good idea, it revealed more info about the bottleneck! When my computer is the iperf3 host and the WRT is the client, the link gets full gigabit speed, but when the WRT is the iperf3 host and the computer is the client it only gets a fraction of that speed. Here's the bottlenecked result over a wired connection with the WRT as the host:

brad@u-turn:~$ date && iperf3 -c cinefile.local
Mon 21 Dec 2020 07:33:41 PM CET
Connecting to host cinefile.local, port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  19.7 MBytes   166 Mbits/sec   23    489 KBytes       
[  5]   1.00-2.00   sec  15.0 MBytes   126 Mbits/sec    0    560 KBytes       
[  5]   2.00-3.00   sec  16.2 MBytes   136 Mbits/sec    0    611 KBytes       
[  5]   3.00-4.00   sec  13.8 MBytes   115 Mbits/sec    3    454 KBytes       
[  5]   4.00-5.00   sec  15.0 MBytes   126 Mbits/sec    0    496 KBytes       
[  5]   5.00-6.00   sec  17.5 MBytes   147 Mbits/sec   25    379 KBytes       
[  5]   6.00-7.00   sec  15.0 MBytes   126 Mbits/sec    0    406 KBytes       
[  5]   7.00-8.00   sec  15.0 MBytes   126 Mbits/sec    0    427 KBytes       
[  5]   8.00-9.00   sec  15.0 MBytes   126 Mbits/sec    0    454 KBytes       
[  5]   9.00-10.00  sec  15.0 MBytes   126 Mbits/sec    0    479 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   157 MBytes   132 Mbits/sec   51             sender
[  5]   0.00-10.03  sec   155 MBytes   130 Mbits/sec                  receiver

iperf Done.

Chadster766 commented 3 years ago

When doing throughtput testing with iPerf3 keep in mind that "-c" means the client is sending to the host and "-R" the host to send to the client. (upload or download from same iPerf3 host)

Mark-GR73 commented 3 years ago

LeBrad, Interesting your nftables ,If you'like post the conf of nftables...

Finnally a solution for IPv6 settings for pppoe with Prefix Delegation from ISP.

Ipv6 Prefix Delegration pppoe.txt

Any additional idea or tip?

Mark-GR73 commented 3 years ago

Hi, Dear Chadster766 i need help for uboot Of course i do not know if here is the right place for this kind of questions.......anyway; I have the 5.6.14 release and not rootfs who i tried but never booted the kernel...

Now, I have two ssds ,one connected to e-sata port and one more to usb3 port. root directory is in esata disk ,which there are two partitions ... -the usb3 has the /var /swap /tmp and /home. mounting from /etc/fstab-

Now what i need...

As primary boot I want to boot McDebian ,to load the .img from esata from 2nd partition and the root dirs from the 1st partition from esata .

As alternative boot i would like the flexibility to boot a wrt image from internal nand ,for easier maintanance purposes..

Please can you advice me? Of course we expecting the next McDebian ultra racing and clever release....

Chadster766 commented 3 years ago

You need a drive connected to the WRT USB 2.0\eSATA port that has the McDebian rootfs and have that operational before you can do anything else.

Mark-GR73 commented 3 years ago

Any wiki or to read something? How the machine will Know at boot time from which device must load the .img file and root dir/s ? I do not need for root-fs edition ,i need it for 5.6.14 ..

Thanks..

Chadster766 commented 3 years ago

Any wiki or to read something? How the machine will Know at boot time from which device must load the .img file and root dir/s ? I do not need for root-fs edition ,i need it for 5.6.14 ..

Thanks..

https://github.com/Chadster766/McDebian/wiki/3.-Make-Root-File-System

Mark-GR73 commented 3 years ago

Thanks for your reply. It is a good start point for that i need. Nice wiki but it is little bit hidden ...on web...

Could i be useful to Mcdebian project someway?

Chadster766 commented 3 years ago

Thanks for your reply. It is a good start point for that i need. Nice wiki but it is little bit hidden ...on web...

Could i be useful to Mcdebian project someway?

Thanks but I'm not sure there is much to do at this point.

Mark-GR73 commented 3 years ago

Success u boot booting to load image from esata. Thanks to your wiki ,and older posts of you.

A nice question...

If the device's nand in this case /dev/mtd5 up to mtd 8 ,are empty ,is there possibility to work as ram ?

Chadster766 commented 3 years ago

It's possible but I wouldn't recommend it.

This type of nand memory can be easily be destroyed by to many write operations where RAM isn't effected in that way.

Mark-GR73 commented 3 years ago

Thanks...

Mark-GR73 commented 3 years ago

Well .. In usb 3 port add a usb3.1 to nvme m2. At boot time the u boot not recognise it at all . Is there something i must know?

Chadster766 commented 3 years ago

Well .. In usb 3 port add a usb3.1 to nvme m2. At boot time the u boot not recognise it at all . Is there something i must know?

You have to have your boot USB key in the USB 2.0 port due to a u-boot limitation.

Your other storage can be connected to the USB 3.0 port.

Mark-GR73 commented 3 years ago

Finally , my previous set up was : esata to sata as root mount, usb3 to sata as /var swap /tmp and /home dir.... A good reason to have two buses divided interworkings and probably higher speed.
And two of them have the almost same speed with hdparm about 700 and 400 M. Working perfectly that way.... -General the thought is that from esata protocol to usb3 in our case , of course the limits are the esata..

-If we try with nvme card ,m2 convertor to usb3 .... With our topology to be : nvme m2 to usb3 as root and sata to esata mount as /var swap /tmp and /home dir....

The results have two different situations ....

u-boot-usb3.log.txt

u-boot-usb3.log2.txt

-Any opinion or advice?

Mark-GR73 commented 3 years ago

It is converter's the problem , it is not possible to read it from any linux machine.. -Pure chinese machines.... It is convertor from nvme m2 to usb 3.1 of course backward compatible with usb3 protocol.

-I will try again but with nvme m2 to usb3 Hardware..

-Anyway...thanks..

Chadster766 / McDebian