Lan interface problem on RPi 3B+

[rstanislav]

Expected Behavior

LAN working

Actual Behavior

Sometimes (about 1/3 or 1/4 chance) LAN interface not working after power on, its in UP state, but packet counter is zero and its not transmitting or receiving any traffic , after interface restart from LUCI or from console (ifdown && ifup) everything starts working.

Steps to Reproduce the Problem

1. Power on router

Specifications

Last version of openmptcp router and VPS.

[Ysurac]

I'm not able to reproduce that. Do you have something in the log when it's not working ?

[rstanislav]

No, nothing strange, i have tested this on multiple rpi 3b+ and results are the same, is there any way to debug this ?

[Ysurac]

By giving me the log I'm asking...

[Ysurac]

Do you use a 2.5A (or more) power source ? No USB on the RPI ? I have also a RPI 3B+. Next version will be updated with latest firmware and patch, but I see nothing that could resolve a lan problem.

[rstanislav]

I have 3 power sources - each is 3A(DC-DC converters) with common ground - 1 for RPI power, and 2 for modems (4 USB Huawei e3372)(power line from RPI usb is cut, and they get power from DC-DC coverters (1 converter for 2 modems), output power set at 5.1v.

I think i have found atleast part of the problem - https://github.com/raspberrypi/linux/issues/2447

How to change kernel config to add options: CONFIG_BRIDGE_VLAN_FILTERING=n CONFIG_VLAN_8021Q=n CONFIG_VLAN_8021Q_GVRP=n ? i'm not 100% sure it will help(because i sometimes see this problem with kevent 4 at start when PRI on table with HDMI connected and if it happens RPI 100% not working). In my case it sometimes still connects to server, so i'm able to connect to it via VPN tunnel and reset LAN interface

[Ysurac]

Only CONFIG_VLAN_8021Q is enabled. Not sure this can change anything...

[rstanislav]

Is it possible to disable it ? Can you tell me how please ? I will test it and report if problem solved. Problem with it is when rpi started with LAN cable attached, and because of bugs in current kernel code of lan78xx driver it can lockup RPI - on table with HDMI attached it looks like nonstop spam of messages .."kevent 4 may have been dropped", according to discussion https://github.com/raspberrypi/linux/issues/2447 user danijelt said that after he disabled it (CONFIG_VLAN_8021Q is main option because it disables vlan support completely) this bug never repeated, at this time according to that discussion its the only way and it is not fixed in any current version on kernel.

sorry for my bad english btw

[Ysurac]

You can follow this: https://github.com/Ysurac/openmptcprouter/wiki/Create-image-for-unsupported-platform using rpi3 as platform and doing a make kernel_menuconfig before the make menuconfig and disable CONFIG_VLAN_8021Q. But I can't disable this, I need this :)

[rstanislav]

Can't build from current sources :( log:

make[3]: Leaving directory '/root/newopen/openmptcprouter/feeds/openmptcprouter/luci-app-omr-bypass'

Host key verification failed. fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists. Makefile:48: recipe for target '/root/newopen/openmptcprouter/rpi3/source/dl/tracebox-0.4.4.tar.bz2' failed make[3]: [/root/newopen/openmptcprouter/rpi3/source/dl/tracebox-0.4.4.tar.bz2] Error 128 make[3]: Leaving directory '/root/newopen/openmptcprouter/feeds/openmptcprouter/tracebox' package/Makefile:107: recipe for target 'package/feeds/openmptcprouter/tracebox/compile' failed make[2]: [package/feeds/openmptcprouter/tracebox/compile] Error 2 make[2]: Leaving directory '/root/newopen/openmptcprouter/rpi3/source' package/Makefile:103: recipe for target '/root/newopen/openmptcprouter/rpi3/source/staging_dir/target-aarch64_cortex-a53_musl/stamp/.package_compile' failed make[1]: *** [/root/newopen/openmptcprouter/rpi3/source/staging_dir/target-aarch64_cortex-a53_musl/stamp/.package_compile] Error 2 make[1]: Leaving directory '/root/newopen/openmptcprouter/rpi3/source' Build failed - please re-run with -j1 to see the real error message /root/newopen/openmptcprouter/rpi3/source/include/toplevel.mk:216: recipe for target 'world' failed

[Ysurac]

Current sources are working, it's build at each push by CircleCI (It's why there is "Passed" in green here https://github.com/Ysurac/openmptcprouter ). Your error seems to be a problem to access github over TLS. You can run with -j1 V=s for more info.

[Ysurac]

As a test, can you try to remove, on OpenMPTCProuter, the line in /etc/init.d/mptcp with ethtool --offload and reboot ?

[rstanislav]

/dl/tracebox-0.4.4.tar.bz2 /root/newopen/openmptcprouter/rpi3/source/dl/ && rm -rf tracebox-0.4.4; ' Checking out files from the git repository... Cloning into 'tracebox-0.4.4'...

The authenticity of host 'github.com (192.30.253.112)' can't be established. RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts. Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

[Ysurac]

TLS problem. No idea why.

I backported a patch for lan78xx (used for ethernet) from the Raspberry Linux Kernel 4.17 branch. I will try to generate an image and test it tomorrow. This may solve some issues.

[rstanislav]

I'm building image on ubuntu as root, problem is not with TLS, but with git client - makefile of package tracebox is configured differently from all other packages and requires git client configured in build system. I have managed to do it by generating ssh\ssl keys, adding them to ssh agent and uploading to github, but i think it should work without all these... Is it possible to change tracebox makefile so it will download release tar.gz package from https://github.com/tracebox/tracebox/archive/v0.4.4.tar.gz so it will not require any of configurating that i have done ?

Also i have managed to build image without vlan support i think - by doing OMR_TARGET="rpi3" ./build.sh -j12 V=s cd rpi3/source

and then i had to copy .config.keep to .config because even after build it was not configured for rpi3 target, but after that there is no CONFIG_VLAN_8021Q option in menuconfig - only mac-vlan, i had to manually edit .config and paste

CONFIG_BRIDGE_VLAN_FILTERING=n CONFIG_VLAN_8021Q=n CONFIG_VLAN_8021Q_GVRP=n

at top of it... and after that build succeeded and i don't see message about vlan hw filtering something at rpi start, that indicates that vlan support disabled i think - but currently don't have time to test, will do in next few hours...

So question is - maybe there is simpler way to include those options for rpi3 target before build... for example if i include them in config-rpi3 file in openmptcprouter root folder (where build.sh is located) ?

BTW i have made a small patch for glorytun-tcp version, so it will accept command line options for local\remote ip and exec custom system file (in my case - bash script with routes) and configure/bring interface up after start without need to manually configure it or by system scripts (this sometimes causes problems because openmptcprouter client device tries to connect very fast, and if glorytun is running, but gtun0 interface is not configured its crashing, and even after glorytun restart OS sometimes configures interface not fast enough so it can crash few times). + i have wrapper script for its startup/restart in case of crash (because systemd scripts are not cross system(only debian) and also can cause problems) i can upload patch file with wrapper script if you want to include it in vps scripts.

[Ysurac]

Tracebox package was changed 2 hours ago :) OpenMPTCProuter can't work without VLAN. An image with patch that may solve many issue on lan is being build now.

What is crashing exactly ? For glorytun patch, this may be send to glorytun upstream: https://github.com/angt/glorytun

[Ysurac]

OpenMPTCProuter v0.31.1 may solve your issue. There is some patch about VLAN on lan78xx, at least this seems to solve some issues for me.

[rstanislav]

Tested without modems(they are mounted in car) - about 30 cold starts - no problems, always working, looks like its fixed, will test more in car with modems during next few days and report back.

Thank you very much for help & support!

[rstanislav]

Tested in car a few times - kevent 4 is not appearing anymore(because i can connect to rpi via glorytun tunnel), but lan interface is still not working correctly - now it have rx counter increasing and zero tx, this happens btw when on lan other device connected - tplink wr902ac with openwrt firmware(for wifi) - i can connect to it and see that on tplink side there is tx increasing and zero rx..

When rpi connected directly to laptop - this never happened (did test about 30 times today). As a workaround i want to write a script that will ping tplink and if no answer will ifdown lan && ifup lan (this fixes problem)... but how i can debug this ?

[Ysurac]

Do you have something in system and kernel log ? (like hw csum failure, dump_backtrace or even many info from omr-tracker)

[rstanislav]

I have found more patches in current and testing linux kernel for rpi(looks like they missing in some kernel versions), can you add them too please ?

https://github.com/raspberrypi/linux/commit/1c70d2de54534eb78728fb405b187a5ec525f7fc https://github.com/raspberrypi/linux/commit/d407fc229cdcd675c0af78513a31e50937ab4e21 https://github.com/raspberrypi/linux/commit/db81c14ce9fbd705c2d3936edecbc6036ace6c05 https://github.com/raspberrypi/linux/commit/c2eb30683b43b13b931bd9cfef6a2a09ac7b7c1e

discussion of this problem is in thread: https://github.com/raspberrypi/linux/issues/2642

[Ysurac]

These patches are already applied in latest version.

[rstanislav]

After many restarts got same problem as in car - https://drive.google.com/open?id=1aDTyeydWDMdPxqjFQX67S1eLakapeTZ4

this prevent rpi from boot, tested on last version 0.31.6

Why openmptcprouter can't work without vlan support ? Can you please help me how to compile it without it, i don't need mac vlan feature, just want to boot it stable :(

[Ysurac]

OpenMPTCProuter can work without vlan support, but you must have a real port by modem. On the raspberry it's possible only by adding USB devices.

I'm not able to reproduce your kevent4 issue. I have an issue when IPv6 is enabled (with an hw csum failure and a call trace dump) and my workaround (ethtool --offload eth0 rx off tx off) is not enabled.

[rstanislav]

i have all modems connected via usb (4 of them), ipv6 i have disabled, if i try make kernel_menuconfig before the make menuconfig after building for rpi3 target(as you mentioned before), i can see that there is vlan option, but target is not configured for pi3, looks like after build it deletes\replaces config by default, and config for rpi3 in named .config.keep but if i rename it to .config and try to kernel_menuconfig/menuconfig i can see that vlan option is missing in kernel, there is only mac-vlan related option for openwrt, and during build it regenerates config if i manually add option to disable vlan and resulting image is still with vlan support.

This bug very rarely occurs when 1 modem connected for example, but with all 4 connected i can see it very often (maybe because they all present in system as ethernet devices (all modem with hilink firmware( and they boot faster than rpi, so rpi see them sometimes even before rpi eth0 module loaded)) - thats cause of this bug i think.

[Ysurac]

You have to disable kmod_macvlan. Config is not regenerated if you make don't run build.sh again.

4 USB modem on a RPI3 is a lot. USB is bad on RPI3 (and lan interface use USB too).

[rstanislav]

even without vlan support in kernel this bug occurs, i have checked commits and looks like main commit that shoud fix this problem is missing, link: https://github.com/raspberrypi/linux/commit/c2eb30683b43b13b931bd9cfef6a2a09ac7b7c1e

Can you add this patch to project please ?

[Ysurac]

It's already applied ( https://github.com/Ysurac/openmptcprouter/blob/master/root/target/linux/brcm2708/patches-4.14/0297-lan78xx-Move-enabling-of-EEE-into-PHY-init-code.patch ). All patches from raspberrypi kernel 4.14 are applied.

[rstanislav]

:((( what else i can do to debug/fix it ? i can reproduce it very easy, and have uart connection to rpi

[Ysurac]

No idea for now... Difficult to tell where is the problem if there is nothing in kernel log.

[rstanislav]

Guy at rpi thread on github said that with latest firmware (start.elf and bootcode.bin from last raspbian) said that after these updates he didn't saw any errors related to kevent4.

openmptcprouter using lede kernel right ? so current version is 4.14.67 ?

[rstanislav]

Is there any way to build 32bit image by using OMR_TARGET="rpi3" ./build.sh command ?

[Ysurac]

OMR_TARGET="rpi2" ./build.sh build the 32bit image for rpi2 & rpi3

[rstanislav]

So finally there is workaround fix for this - tested and so far no problems.(descriptions why it fixes it in that thread) link to patch: https://github.com/raspberrypi/linux/issues/2447#issuecomment-422007369

Can you please add it to patches in openmptcprouter project ?

[Ysurac]

This will be available in OpenMPTCProuter v0.36.2 (today or tomorrow)

[Ysurac]

OpenMPTCProuter v0.36.2 with this patch is available.