Network Disconnect cannot be reversed on Raspbian

abirger commented 3 years ago

Clicking on "Network Disconnect" button indeed shuts down the device-facing network adapter (eth1). However, clicking on the "Network Reconnect" button does not bring it back up despite the message that the changes were applied successfully. ifconfig does not show eth1 anymore, packets are not going through, and it cannot be fixed by anything other than reboot.

Have not checked that feature on Armbian, maybe it works there.

andni233 commented 3 years ago

I'll take a look at see if I can reproduce this issue.

To help troubleshoot this some additional info might help:

What Znail image are you running?
What hardware are you using?
Could you share the kernel log from the device (output from dmesg) and the Znail log (output from journalctl -u znail)?

abirger commented 2 years ago

@andni233, I use Raspberry Pi 3 Model B Rev 1.2 with the "raspbian_image_2021-08-29-znail-lite" posted under release v0.6.0. The only difference from the release is that I modified the "update-network-interfaces" file to assign a static IP to br0. I attached the modified file along with the logs.

dmesg.log ifconfig.log znail.log update-network-interfaces.txt

Risca commented 2 years ago

I find that this page contains some useful HW info on the different Raspberry Pi models: https://github.com/mvp/uhubctl#raspberry-pi-b2b3b

Might possibly provide a clue on what's going on.

I've got quite a busy schedule the coming days, so we'll see if I have time to testing out this bug. I think I should have a model 3 laying around here somewhere...

Risca commented 2 years ago

I took another closer look on the kernel log, and I can see that it looks like a USB network interface do show up again:

[   99.404489] usb 1-1.4: new high-speed USB device number 5 using dwc_otg
[   99.547974] usb 1-1.4: New USB device found, idVendor=0b95, idProduct=7720, bcdDevice= 0.01
[   99.547999] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   99.548014] usb 1-1.4: Product: AX88772A
[   99.548029] usb 1-1.4: Manufacturer: ASIX Elec. Corp.
[   99.548043] usb 1-1.4: SerialNumber: 000387
[  100.014969] asix 1-1.4:1.0 eth1: register 'asix' at usb-3f980000.usb-1.4, ASIX AX88772 USB 2.0 Ethernet, 00:0e:c6:78:bf:54

However, the ifconfig log does not list it :thinking: weird. Have you tried to unplug and plug in the USB network card after clicking the "Network reconnect" button? This won't be a solution, but will give some information on how to debug this issue.

Lastly, I saw this early in your kernel log:

[    9.354552] Under-voltage detected! (0x00050005)

Have you tried using a different power supply?

abirger commented 2 years ago

@Risca, I replaced the power supply but still cannot reconnect the eth1 port via Web GUI. Unplugging and plugging in of the USB adapter did not help either.

You are correct, the Reconnect command actually enables USB adapter but for some reason, the eth1 port does not come up (which is correctly reflected by ifconfig output). I can force it up with either 'sudo ip link set eth1 up' or 'sudo ifconfig eth1 up' command; however, it does not restore the bridge functionality:

[  169.139008] asix 1-1.4:1.0 eth1: link up, 100Mbps, full-duplex, lpa 0xCDE1
[  169.152412] asix 1-1.4:1.0 eth1: link up, 100Mbps, full-duplex, lpa 0xCDE1

When I try to restart networking, the command 'sudo /etc/init.d/networking restart' fails

[....] Restarting networking (via systemctl): networking.serviceJob for networking.service failed because the control process exited with error code.
See "systemctl status networking.service" and "journalctl -xe" for details.
 failed!

However, the bridge comes back to life despite the restart failure:

[  765.797735] br0: port 2(eth1) entered blocking state
[  765.797750] br0: port 2(eth1) entered disabled state
[  765.798271] device eth1 entered promiscuous mode
[  765.798355] br0: port 2(eth1) entered blocking state
[  765.798363] br0: port 2(eth1) entered forwarding state

Maybe the UGREEN USB network card has certain issues with the driver. I do not have another card from a different vendor to check.

Risca commented 2 years ago

This is the code that handles the re-connect call:

<cut>
    _usb.enable_all_usb_ports()
    while not self._poll_network_interface("eth1"):
        time.sleep(0.1)
    self.tc.apply(self.tc.disciplines)

def _poll_network_interface(self, name):
    if os.getenv("ZNAIL_FORCE_INTERFACE_UP", False):
        return True
    return name in os.listdir("/sys/class/net/")

where tc.apply() is wrapping calls to the tc command line utility. From what I could tell, the ZNAIL_FORCE_INTERFACE_UP environment variable should not be set, so it should poll /sys/class/net/ to see if the network interface shows up there.

@abirger, could you check if "eth1" show up in /sys/class/net/ after reconnecting?

Maybe the kernel has changed some names there and we're not reacting to it as we should :thinking:

EDIT: There could be some timing issues as well. I.e., we have to wait a bit longer for some adapters to work correctly before we can issue any tc commands :thinking: There might be some other criteria to look for to know when an adapter is ready to be configured? @andni233, any ideas?

abirger commented 2 years ago

@Risca , the "eth1" seems identical tome before and after disconnect/reconnect operation.

Before disconnecting:

pi@bridge:~ $ ls -halt /sys/class/net/
total 0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 eth1 -> ../../devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4/1-1.4:1.0/net/eth1
lrwxrwxrwx  1 root root 0 Oct 27 07:43 br0 -> ../../devices/virtual/net/br0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 wlan0 -> ../../devices/platform/soc/3f300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
drwxr-xr-x  2 root root 0 Oct 27 07:43 .
lrwxrwxrwx  1 root root 0 Oct 27 07:43 eth0 -> ../../devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1:1.0/net/eth0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 lo -> ../../devices/virtual/net/lo
drwxr-xr-x 54 root root 0 Feb 14  2019 ..

After disconnecting:

pi@bridge:~ $ ls -halt /sys/class/net/
total 0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 br0 -> ../../devices/virtual/net/br0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 wlan0 -> ../../devices/platform/soc/3f300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
drwxr-xr-x  2 root root 0 Oct 27 07:43 .
lrwxrwxrwx  1 root root 0 Oct 27 07:43 eth0 -> ../../devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1:1.0/net/eth0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 lo -> ../../devices/virtual/net/lo
drwxr-xr-x 54 root root 0 Feb 14  2019 ..

After reconnecting:

pi@bridge:~ $ ls -halt /sys/class/net/
total 0
lrwxrwxrwx  1 root root 0 Oct 27 07:44 eth1 -> ../../devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4/1-1.4:1.0/net/eth1
lrwxrwxrwx  1 root root 0 Oct 27 07:43 br0 -> ../../devices/virtual/net/br0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 wlan0 -> ../../devices/platform/soc/3f300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
drwxr-xr-x  2 root root 0 Oct 27 07:43 .
lrwxrwxrwx  1 root root 0 Oct 27 07:43 eth0 -> ../../devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1:1.0/net/eth0
lrwxrwxrwx  1 root root 0 Oct 27 07:43 lo -> ../../devices/virtual/net/lo
drwxr-xr-x 54 root root 0 Feb 14  2019 ..

znailnetem / znail

Network Disconnect cannot be reversed on Raspbian #10