search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.17k stars 5.01k forks source link

SPI Ethernet (enc28j60) does not work when added to a bridge #4488

Open AndysRobotics opened 3 years ago

AndysRobotics commented 3 years ago

Describe the bug When adding enc28j60 to a pi and creating a bridge, there appears to be a discrepancy of 4 bytes when traffic is crossing the bridge and SPI eth port. This causes full size packets to get dropped that are traversing the bridge due to being oversized. This is only an issue when the SPI ethernet port is added to a bridge, when running independently it works as expected. Adding a USB ethernet port into the same bridge works flawlessly the issue is only when an SPI ethernet port is added to a bridge.

To reproduce Hardware required

Using the image (2021-05-07-raspios-buster-armhf-lite) enable enc28j60 driver and wire into SPI port. Add USB eth port (such as Rankie gigabit ethernet adaptor) to help test bridge

To enable enc28j60 add following to /boot/config.txt

[pi4]
dtoverlay=spi0-1cs,cs0_pin=8
dtoverlay=enc28j60,int_pin=22,speed=12500000
[all]

To enable bridge sudo apt-get install bridge-utils and add following to /etc/network/interfaces.d/br0

auto br0
iface br0 inet static
        bridge_ports eth0 eth1 eth2
        bridge_stp off
        bridge_waitport 0
        bridge_fd 0
        address 10.0.1.106
        netmask 255.255.255.0
        gateway 10.0.1.1

Disable DHCP client on all ports add denyinterfaces eth0 eth1 eth2 to top of /etc/dhcpcd.conf to prevent ethernet interfaces obtaining an address. Then reboot pi.

Connect host pi (with bridge and additional ports) to network via built in port. Add second pi (client) with default image and dhcp to USB ethernet port. Add third pi (client) with default image and dhcp to SPI ethernet port.

Ping all devices from another device connected to same network with varying packet sizes (1468, 1472, 1473) with don't defrag set. ping 10.0.1.xxx -t -l 1468 -f from windows or ping 10.0.1.xxx -s 1468 -M do from linux

Expected behaviour Host to have static address assigned to bridge br0. Clients to pick up DHCP. All devices to respond with correct sizes when 1468 or 1472 sized packet is sent and error with fragment warning when 1473 bytes sent.

Actual behaviour Host pi and the client pi on USB port both behave as expected. Pi on SPI port gets request timed out on 1469-1472 bytes and then fragment warning when 1473 bytes sent This causes anything that needs to transfer larger than 1468 bytes at once to fail, so sftp and web hosting with larger files to fail from the pi connected to the SPI port.

Additional context On the host pi, eth0 is built in port, eth1 is USB port and eth2 is SPI port. Swapping the two clients around keeps the issue with the SPI port only. The host can ping the client on the SPI port at 1472 and the client on the SPI port can ping the host at 1472, the issue only occurs when the client has to cross the bridge.

Setting the MTU on the client connected in the SPI port to 1496 allows pings up to 1468 and will warn about ping 1469 and larger. This indicates that path discovery is working but we have devices that cannot change their MTU so this isn't a solution.

Changing the MTU value (up or down) on the bridge interface or on any of the routes/interfaces on the host pi has no effect. This indicates the issue is with traffic inbound to the SPI ethernet port.

On our production we are using 2x enc28j60 on SPI0 and SPI1 so iptables or other forms of bridging is not available. The issue is present when using 1 or 2 enc28j60 modules.

System System Information

Raspberry Pi 4 Model B Rev 1.4 PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)"

Raspberry Pi reference 2021-05-07 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, dcfd74d7d1fa293065ac6d565711e9ff891fe2b8, stage2

Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux Revision : b03114 Serial : 100000005aaf6c59 Model : Raspberry Pi 4 Model B Rev 1.4 Throttled flag : throttled=0x0 Camera : supported=0 detected=0

Videocore information

Apr 30 2021 13:45:52 Copyright (c) 2012 Broadcom version d7f29d96450abfc77cd6cf011af1faf1e03e5e56 (clean) (release) (start)

alloc failures: 0 compactions: 0 legacy block fails: 0

Filesystem information

Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 1507280 1291520 121144 92% / devtmpfs 823564 0 823564 0% /dev tmpfs 955660 0 955660 0% /dev/shm tmpfs 955660 8612 947048 1% /run tmpfs 5120 4 5116 1% /run/lock tmpfs 955660 0 955660 0% /sys/fs/cgroup /dev/mmcblk0p1 258095 48782 209313 19% /boot tmpfs 191132 0 191132 0% /run/user/1000

Filename Type Size Used Priority /var/swap file 102396 0 -2

Package version information

raspberrypi-ui-mods: Installed: (none) raspberrypi-sys-mods: Installed: 20210310 openbox: Installed: (none) lxpanel: Installed: (none) pcmanfm: Installed: (none) rpd-plym-splash: Installed: (none)

Networking Information

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet x.x.x.x netmask x.x.x.x broadcast x.x.x.x inet6 y::y.y.y.y prefixlen 64 scopeid 0x20 ether m.m.m.m txqueuelen 1000 (Ethernet) RX packets 461 bytes 79102 (77.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 897 bytes 169492 (165.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet x.x.x.x netmask x.x.x.x broadcast x.x.x.x ether m.m.m.m txqueuelen 1000 (Ethernet) RX packets 504 bytes 98800 (96.4 KiB) RX errors 0 dropped 2 overruns 0 frame 0 TX packets 916 bytes 171404 (167.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether m.m.m.m txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether m.m.m.m txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 65

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet x.x.x.x netmask x.x.x.x inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

USB Information

/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M | Port 2: Dev 2, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M | Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

config.txt

arm_freq=1500 audio_pwm_mode=514 config_hdmi_boost=5 core_freq=500 core_freq_min=200 disable_commandline_tags=2 disable_l2cache=1 display_hdmi_rotate=-1 display_lcd_rotate=-1 dvfs=2 enable_gic=1 force_eeprom_read=1 force_pwm_open=1 framebuffer_ignore_alpha=1 framebuffer_swap=1 gpu_freq=500 gpu_freq_min=250 init_uart_clock=0x2dc6c00 lcd_framerate=60 mask_gpu_interrupt0=1024 mask_gpu_interrupt1=0x10000 max_framebuffers=2 over_voltage_avs=-20000 pause_burst_frames=1 program_serial_random=1 total_mem=2048 hdmi_force_cec_address:0=65535 hdmi_force_cec_address:1=65535 hdmi_pixel_freq_limit:0=0x11e1a300 hdmi_pixel_freq_limit:1=0x11e1a300 device_tree=- overlay_prefix=overlays/ hdmi_cvt:0= hdmi_cvt:1= hdmi_edid_filename:0= hdmi_edid_filename:1= hdmi_timings:0= hdmi_timings:1=

cmdline.txt

coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=E4:5F:01:33:AF:C4 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=tty1 root=PARTUUID=9730496b-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet

raspi-gpio settings

BANK0 (GPIO 0 to 27): GPIO 0: level=1 fsel=0 func=INPUT pull=UP GPIO 1: level=1 fsel=0 func=INPUT pull=UP GPIO 2: level=1 fsel=0 func=INPUT pull=UP GPIO 3: level=1 fsel=0 func=INPUT pull=UP GPIO 4: level=1 fsel=0 func=INPUT pull=UP GPIO 5: level=1 fsel=0 func=INPUT pull=UP GPIO 6: level=1 fsel=0 func=INPUT pull=UP GPIO 7: level=1 fsel=0 func=INPUT pull=UP GPIO 8: level=1 fsel=1 func=OUTPUT pull=UP GPIO 9: level=0 fsel=4 alt=0 func=SPI0_MISO pull=DOWN GPIO 10: level=0 fsel=4 alt=0 func=SPI0_MOSI pull=DOWN GPIO 11: level=0 fsel=4 alt=0 func=SPI0_SCLK pull=DOWN GPIO 12: level=1 fsel=0 func=INPUT pull=DOWN GPIO 13: level=0 fsel=0 func=INPUT pull=DOWN GPIO 14: level=1 fsel=0 func=INPUT pull=NONE GPIO 15: level=1 fsel=0 func=INPUT pull=UP GPIO 16: level=0 fsel=0 func=INPUT pull=DOWN GPIO 17: level=1 fsel=0 func=INPUT pull=DOWN GPIO 18: level=1 fsel=0 func=INPUT pull=DOWN GPIO 19: level=0 fsel=0 func=INPUT pull=DOWN GPIO 20: level=0 fsel=0 func=INPUT pull=DOWN GPIO 21: level=0 fsel=0 func=INPUT pull=DOWN GPIO 22: level=1 fsel=0 func=INPUT pull=NONE GPIO 23: level=1 fsel=0 func=INPUT pull=DOWN GPIO 24: level=0 fsel=0 func=INPUT pull=DOWN GPIO 25: level=1 fsel=0 func=INPUT pull=DOWN GPIO 26: level=0 fsel=0 func=INPUT pull=DOWN GPIO 27: level=1 fsel=0 func=INPUT pull=DOWN BANK1 (GPIO 28 to 45): GPIO 28: level=1 fsel=2 alt=5 func=RGMII_MDIO pull=UP GPIO 29: level=0 fsel=2 alt=5 func=RGMII_MDC pull=DOWN GPIO 30: level=0 fsel=7 alt=3 func=CTS0 pull=UP GPIO 31: level=0 fsel=7 alt=3 func=RTS0 pull=NONE GPIO 32: level=1 fsel=7 alt=3 func=TXD0 pull=NONE GPIO 33: level=1 fsel=7 alt=3 func=RXD0 pull=UP GPIO 34: level=0 fsel=7 alt=3 func=SD1_CLK pull=NONE GPIO 35: level=1 fsel=7 alt=3 func=SD1_CMD pull=UP GPIO 36: level=1 fsel=7 alt=3 func=SD1_DAT0 pull=UP GPIO 37: level=1 fsel=7 alt=3 func=SD1_DAT1 pull=UP GPIO 38: level=1 fsel=7 alt=3 func=SD1_DAT2 pull=UP GPIO 39: level=1 fsel=7 alt=3 func=SD1_DAT3 pull=UP GPIO 40: level=0 fsel=4 alt=0 func=PWM1_0 pull=NONE GPIO 41: level=0 fsel=4 alt=0 func=PWM1_1 pull=NONE GPIO 42: level=0 fsel=1 func=OUTPUT pull=UP GPIO 43: level=1 fsel=0 func=INPUT pull=UP GPIO 44: level=1 fsel=0 func=INPUT pull=UP GPIO 45: level=1 fsel=0 func=INPUT pull=UP BANK2 (GPIO 46 to 53): GPIO 46: level=0 fsel=0 func=INPUT pull=UP GPIO 47: level=0 fsel=0 func=INPUT pull=UP GPIO 48: level=0 fsel=0 func=INPUT pull=DOWN GPIO 49: level=0 fsel=0 func=INPUT pull=DOWN GPIO 50: level=0 fsel=0 func=INPUT pull=DOWN GPIO 51: level=0 fsel=0 func=INPUT pull=DOWN GPIO 52: level=0 fsel=0 func=INPUT pull=DOWN GPIO 53: level=0 fsel=0 func=INPUT pull=DOWN

vcdbg log messages

debug_sym: vc_mem_copy: Unable to open '/dev/fb0': No such file or directory(2)

003418.640: arasan: arasan_emmc_open 003578.818: brfs: File read: /mfs/sd/config.txt 003579.662: brfs: File read: 2004 bytes 003644.181: brfs: File read: /mfs/sd/config.txt 003660.613: brfs: File read: 2004 bytes 004154.743: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined 004157.240: *** Restart logging 004161.122: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead 004164.553: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead 004164.566: HDMI0: hdmi_pixel_encoding: 300000000 004164.580: HDMI1: hdmi_pixel_encoding: 300000000 004169.326: dtb_file 'bcm2711-rpi-4-b.dtb' 004174.437: brfs: File read: /mfs/sd/bcm2711-rpi-4-b.dtb 004174.455: Loading 'bcm2711-rpi-4-b.dtb' to 0x100 size 0xbfc2 004186.913: brfs: File read: 49090 bytes 004197.080: brfs: File read: /mfs/sd/overlays/overlay_map.dtb 004259.435: brfs: File read: 1523 bytes 004261.506: brfs: File read: /mfs/sd/config.txt 004262.326: brfs: File read: 2004 bytes 004272.893: brfs: File read: /mfs/sd/overlays/spi0-1cs.dtbo 004284.813: Loaded overlay 'spi0-1cs' 004284.824: dtparam: cs0_pin=8 004301.891: brfs: File read: 1045 bytes 004305.713: brfs: File read: /mfs/sd/overlays/enc28j60.dtbo 004314.282: Loaded overlay 'enc28j60' 004314.294: dtparam: int_pin=22 004314.742: dtparam: speed=20000000 004315.047: dtparam: audio=on 004350.654: brfs: File read: 1403 bytes 004363.293: brfs: File read: /mfs/sd/overlays/vc4-fkms-v3d.dtbo 004380.440: Loaded overlay 'vc4-fkms-v3d' 004422.576: brfs: File read: 1446 bytes 004424.187: brfs: File read: /mfs/sd/cmdline.txt 004424.222: Read command line from file 'cmdline.txt': 004424.235: 'console=tty1 root=PARTUUID=9730496b-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet' 005449.972: brfs: File read: 104 bytes 006004.524: brfs: File read: /mfs/sd/kernel7l.img 006004.544: Loading 'kernel7l.img' to 0x8000 size 0x662680 006004.571: Device tree loaded to 0x2eff3a00 (size 0xc588) 006010.689: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined 008610.092: vchiq_core: vchiq_init_state: slot_zero = 0xded80000, is_master = 1

dmesg log

[ 0.000000] Booting Linux on physical CPU 0x0 [ 0.000000] Linux version 5.10.17-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1414 SMP Fri Apr 30 13:20:47 BST 2021 [ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d [ 0.000000] CPU: div instructions available: patching division code [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache [ 0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.4 [ 0.000000] Memory policy: Data cache writealloc [ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ec00000, size 256 MiB [ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff] [ 0.000000] node 0: [mem 0x0000000040000000-0x000000007fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000007fffffff] [ 0.000000] On node 0 totalpages: 504832 [ 0.000000] DMA zone: 2304 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 196608 pages, LIFO batch:63 [ 0.000000] HighMem zone: 308224 pages, LIFO batch:63 [ 0.000000] percpu: Embedded 20 pages/cpu s50700 r8192 d23028 u81920 [ 0.000000] pcpu-alloc: s50700 r8192 d23028 u81920 alloc=20*4096 [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 502528 [ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=m.m.m.m vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=tty1 root=PARTUUID=9730496b-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet [ 0.000000] Kernel parameter elevator= does not have any effect anymore. Please use sysfs to set IO scheduler for individual devices. [ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear) [ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] software IO TLB: mapped [mem 0x0000000019400000-0x000000001d400000] (64MB) [ 0.000000] Memory: 1647132K/2019328K available (10240K kernel code, 1358K rwdata, 3160K rodata, 2048K init, 890K bss, 110052K reserved, 262144K cma-reserved, 1232896K highmem) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] ftrace: allocating 33942 entries in 67 pages [ 0.000000] ftrace: allocated 67 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies. [ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16 [ 0.000000] GIC: Using split EOI/Deactivate mode [ 0.000000] random: get_random_bytes called from start_kernel+0x3c8/0x59c with crng_init=0 [ 0.000008] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns [ 0.000026] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns [ 0.000087] bcm2835: system timer (irq = 25) [ 0.000742] arch_timer: cp15 timer(s) running at 54.00MHz (phys). [ 0.000760] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns [ 0.000778] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns [ 0.000793] Switching to timer-based delay loop, resolution 18ns [ 0.001046] Console: colour dummy device 80x30 [ 0.001112] printk: console [tty1] enabled [ 0.001165] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=540000) [ 0.001198] pid_max: default: 32768 minimum: 301 [ 0.001366] LSM: Security Framework initializing [ 0.001552] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear) [ 0.001578] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear) [ 0.003157] Disabling memory control group subsystem [ 0.003278] CPU: Testing write buffer coherency: ok [ 0.003738] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.004950] Setting up static identity map for 0x200000 - 0x20003c [ 0.005148] rcu: Hierarchical SRCU implementation. [ 0.006077] smp: Bringing up secondary CPUs ... [ 0.007298] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001 [ 0.008676] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002 [ 0.009978] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003 [ 0.010135] smp: Brought up 1 node, 4 CPUs [ 0.010156] SMP: Total of 4 processors activated (432.00 BogoMIPS). [ 0.010171] CPU: All CPU(s) started in HYP mode. [ 0.010185] CPU: Virtualization extensions available. [ 0.011112] devtmpfs: initialized [ 0.025074] VFP support v0.3: implementor 41 architecture 3 part 40 variant 8 rev 0 [ 0.025308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.025339] futex hash table entries: 1024 (order: 4, 65536 bytes, linear) [ 0.032338] pinctrl core: initialized pinctrl subsystem [ 0.033405] NET: Registered protocol family 16 [ 0.037461] DMA: preallocated 1024 KiB pool for atomic coherent allocations [ 0.038222] audit: initializing netlink subsys (disabled) [ 0.038477] audit: type=2000 audit(0.030:1): state=initialized audit_enabled=0 res=1 [ 0.039055] thermal_sys: Registered thermal governor 'step_wise' [ 0.039748] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers. [ 0.039765] hw-breakpoint: maximum watchpoint size is 8 bytes. [ 0.040160] Serial: AMBA PL011 UART driver [ 0.073305] bcm2835-mbox fe00b880.mailbox: mailbox enabled [ 0.091169] raspberrypi-firmware soc:firmware: Attached to firmware from 2021-04-30T13:45:52, variant start [ 0.101181] raspberrypi-firmware soc:firmware: Firmware hash is d7f29d96450abfc77cd6cf011af1faf1e03e5e56 [ 0.146593] bcm2835-dma fe007000.dma: DMA legacy API manager, dmachans=0x1 [ 0.150739] vgaarb: loaded [ 0.151178] SCSI subsystem initialized [ 0.151796] usbcore: registered new interface driver usbfs [ 0.151855] usbcore: registered new interface driver hub [ 0.151929] usbcore: registered new device driver usb [ 0.152282] usb_phy_generic phy: supply vcc not found, using dummy regulator [ 0.154159] clocksource: Switched to clocksource arch_sys_counter [ 1.152400] VFS: Disk quotas dquot_6.6.0 [ 1.152507] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) [ 1.152670] FS-Cache: Loaded [ 1.152880] CacheFiles: Loaded [ 1.162987] NET: Registered protocol family 2 [ 1.163814] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear) [ 1.163867] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear) [ 1.164010] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 1.164080] TCP: Hash tables configured (established 8192 bind 8192) [ 1.164297] UDP hash table entries: 512 (order: 2, 16384 bytes, linear) [ 1.164333] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear) [ 1.164621] NET: Registered protocol family 1 [ 1.165329] RPC: Registered named UNIX socket transport module. [ 1.165344] RPC: Registered udp transport module. [ 1.165360] RPC: Registered tcp transport module. [ 1.165374] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.165398] PCI: CLS 0 bytes, default 64 [ 1.168297] Initialise system trusted keyrings [ 1.168533] workingset: timestamp_bits=14 max_order=19 bucket_order=5 [ 1.177136] zbud: loaded [ 1.179027] FS-Cache: Netfs 'nfs' registered for caching [ 1.179788] NFS: Registering the id_resolver key type [ 1.179847] Key type id_resolver registered [ 1.179862] Key type id_legacy registered [ 1.180000] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 1.181037] Key type asymmetric registered [ 1.181054] Asymmetric key parser 'x509' registered [ 1.181244] bounce: pool size: 64 pages [ 1.181288] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 1.181495] io scheduler mq-deadline registered [ 1.181511] io scheduler kyber registered [ 1.185999] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges: [ 1.186030] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff] [ 1.186110] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000 [ 1.186199] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x007fffffff -> 0x0400000000 [ 1.246266] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC) [ 1.246664] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00 [ 1.246684] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.246706] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff]) [ 1.246794] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400 [ 1.247039] pci 0000:00:00.0: PME# supported from D0 D3hot [ 1.250550] PCI: bus0: Fast back to back transfers disabled [ 1.250922] pci 0000:01:00.0: [1106:3483] type 00 class 0x0c0330 [ 1.251049] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00000fff 64bit] [ 1.251527] pci 0000:01:00.0: PME# supported from D0 D3hot [ 1.255055] PCI: bus1: Fast back to back transfers disabled [ 1.255118] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff] [ 1.255142] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600000fff 64bit] [ 1.255226] pci 0000:00:00.0: PCI bridge to [bus 01] [ 1.255251] pci 0000:00:00.0: bridge window [mem 0x600000000-0x6000fffff] [ 1.255658] pcieport 0000:00:00.0: enabling device (0140 -> 0142) [ 1.255873] pcieport 0000:00:00.0: PME: Signaling with IRQ 63 [ 1.264315] iproc-rng200 fe104000.rng: hwrng registered [ 1.264623] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB) [ 1.265487] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000 [ 1.277765] brd: module loaded [ 1.290107] loop: module loaded [ 1.291762] Loading iSCSI transport class v2.0-870. [ 1.294235] libphy: Fixed MDIO Bus: probed [ 1.295924] bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000 [ 1.314203] libphy: bcmgenet MII bus: probed [ 1.404304] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus [ 1.405455] usbcore: registered new interface driver r8152 [ 1.405534] usbcore: registered new interface driver lan78xx [ 1.405595] usbcore: registered new interface driver smsc95xx [ 1.453517] xhci_hcd 0000:01:00.0: xHCI Host Controller [ 1.453555] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1 [ 1.456835] xhci_hcd 0000:01:00.0: hcc params 0x002841eb hci version 0x100 quirks 0x0000030000000890 [ 1.458164] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10 [ 1.458184] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.458201] usb usb1: Product: xHCI Host Controller [ 1.458218] usb usb1: Manufacturer: Linux 5.10.17-v7l+ xhci-hcd [ 1.458234] usb usb1: SerialNumber: 0000:01:00.0 [ 1.458994] hub 1-0:1.0: USB hub found [ 1.459104] hub 1-0:1.0: 1 port detected [ 1.459776] xhci_hcd 0000:01:00.0: xHCI Host Controller [ 1.459815] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2 [ 1.459840] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed [ 1.460340] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10 [ 1.460359] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.460376] usb usb2: Product: xHCI Host Controller [ 1.460393] usb usb2: Manufacturer: Linux 5.10.17-v7l+ xhci-hcd [ 1.460409] usb usb2: SerialNumber: 0000:01:00.0 [ 1.461114] hub 2-0:1.0: USB hub found [ 1.461185] hub 2-0:1.0: 4 ports detected [ 1.462834] dwc_otg: version 3.00a 10-AUG-2012 (platform bus) [ 1.463155] dwc_otg: FIQ enabled [ 1.463170] dwc_otg: NAK holdoff enabled [ 1.463184] dwc_otg: FIQ split-transaction FSM enabled [ 1.463202] Module dwc_common_port init [ 1.463653] usbcore: registered new interface driver uas [ 1.463760] usbcore: registered new interface driver usb-storage [ 1.463970] mousedev: PS/2 mouse device common for all mice [ 1.465675] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer [ 1.469175] sdhci: Secure Digital Host Controller Interface driver [ 1.469190] sdhci: Copyright(c) Pierre Ossman [ 1.469894] mmc-bcm2835 fe300000.mmcnr: could not get clk, deferring probe [ 1.470515] sdhci-pltfm: SDHCI platform and OF driver helper [ 1.474667] ledtrig-cpu: registered to indicate activity on CPUs [ 1.475025] hid: raw HID events driver (C) Jiri Kosina [ 1.475235] usbcore: registered new interface driver usbhid [ 1.475250] usbhid: USB HID core driver [ 1.480987] Initializing XFRM netlink socket [ 1.481030] NET: Registered protocol family 17 [ 1.481147] Key type dns_resolver registered [ 1.481499] Registering SWP/SWPB emulation handler [ 1.481683] registered taskstats version 1 [ 1.481710] Loading compiled-in X.509 certificates [ 1.482519] Key type ._fscrypt registered [ 1.482535] Key type .fscrypt registered [ 1.482550] Key type fscrypt-provisioning registered [ 1.494077] uart-pl011 fe201000.serial: there is not valid maps for state default [ 1.494439] uart-pl011 fe201000.serial: cts_event_workaround enabled [ 1.494517] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 36, base_baud = 0) is a PL011 rev2 [ 1.501236] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver [ 1.502290] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0 [ 1.502307] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated [ 1.531743] of_cfs_init [ 1.532008] of_cfs_init: OK [ 1.552074] mmc1: queuing unknown CIS tuple 0x80 (2 bytes) [ 1.553766] mmc1: queuing unknown CIS tuple 0x80 (3 bytes) [ 1.555456] mmc1: queuing unknown CIS tuple 0x80 (3 bytes) [ 1.558478] mmc1: queuing unknown CIS tuple 0x80 (7 bytes) [ 1.560170] mmc1: queuing unknown CIS tuple 0x80 (3 bytes) [ 1.566873] mmc0: SDHCI controller on fe340000.emmc2 [fe340000.emmc2] using ADMA [ 1.567566] Waiting for root device PARTUUID=9730496b-02... [ 1.608081] random: fast init done [ 1.639904] mmc1: new high speed SDIO card at address 0001 [ 1.675840] mmc0: new ultra high speed DDR50 SDHC card at address aaaa [ 1.676855] mmcblk0: mmc0:aaaa SD32G 29.7 GiB [ 1.679709] mmcblk0: p1 p2 [ 1.691960] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null) [ 1.692030] VFS: Mounted root (ext4 filesystem) readonly on device 179:2. [ 1.696981] devtmpfs: mounted [ 1.705933] Freeing unused kernel memory: 2048K [ 1.754524] Run /sbin/init as init process [ 1.754539] with arguments: [ 1.754553] /sbin/init [ 1.754568] with environment: [ 1.754582] HOME=/ [ 1.754596] TERM=linux [ 1.834216] usb 1-1: new high-speed USB device number 2 using xhci_hcd [ 2.016909] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21 [ 2.016931] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 2.016949] usb 1-1: Product: USB2.0 Hub [ 2.018992] hub 1-1:1.0: USB hub found [ 2.019299] hub 1-1:1.0: 4 ports detected [ 2.116081] systemd[1]: System time before build time, advancing clock. [ 2.164469] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd [ 2.205853] usb 2-2: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00 [ 2.205875] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=6 [ 2.205893] usb 2-2: Product: USB 10/100/1000 LAN [ 2.205910] usb 2-2: Manufacturer: Realtek [ 2.205926] usb 2-2: SerialNumber: 000001000000 [ 2.214886] NET: Registered protocol family 10 [ 2.216257] Segment Routing with IPv6 [ 2.268291] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid) [ 2.269549] systemd[1]: Detected architecture arm. [ 2.365265] systemd[1]: Set hostname to . [ 2.385007] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd [ 2.431260] r8152 2-2:1.0: Direct firmware load for rtl_nic/rtl8153a-3.fw failed with error -2 [ 2.431291] r8152 2-2:1.0: unable to load firmware patch rtl_nic/rtl8153a-3.fw (-2) [ 2.476138] r8152 2-2:1.0 eth1: v1.11.11 [ 2.965380] random: systemd: uninitialized urandom read (16 bytes read) [ 2.974960] random: systemd: uninitialized urandom read (16 bytes read) [ 2.975054] systemd[1]: Reached target Swap. [ 2.975504] random: systemd: uninitialized urandom read (16 bytes read) [ 2.975844] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 2.976616] systemd[1]: Listening on Syslog Socket. [ 2.977248] systemd[1]: Listening on udev Kernel Socket. [ 2.978177] systemd[1]: Listening on Journal Audit Socket. [ 2.978611] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 2.978746] systemd[1]: Reached target Local Encrypted Volumes. [ 3.484537] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ 3.632708] systemd-journald[113]: Received request to flush runtime journal from PID 1 [ 4.119153] rpivid-mem feb00000.hevc-decoder: rpivid-hevcmem initialised: Registers at 0xfeb00000 length 0x00010000 [ 4.120600] rpivid-mem feb10000.rpivid-local-intc: rpivid-intcmem initialised: Registers at 0xfeb10000 length 0x00001000 [ 4.121178] rpivid-mem feb20000.h264-decoder: rpivid-h264mem initialised: Registers at 0xfeb20000 length 0x00010000 [ 4.122299] rpivid-mem feb30000.vp9-decoder: rpivid-vp9mem initialised: Registers at 0xfeb30000 length 0x00010000 [ 4.207538] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned. [ 4.213999] mc: Linux media interface: v0.10 [ 4.217403] bcm2835_vc_sm_cma_probe: Videocore shared memory driver

[ 4.221781] [vc_sm_connected_init]: installed successfully [ 4.255941] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned. [ 4.301113] bcm2835_audio bcm2835_audio: card created with 8 channels [ 4.358947] videodev: Linux video capture interface: v2.00 [ 4.394423] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned. [ 4.396087] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned. [ 4.398002] [drm] No displays found. Consider forcing hotplug if HDMI is attached [ 4.398135] vc4-drm gpu: bound fe600000.firmwarekms (ops vc4_fkms_ops [vc4]) [ 4.398575] [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 0 [ 4.409909] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned. [ 4.413750] bcm2835_isp: module is from the staging directory, the quality is unknown, you have been warned. [ 4.414299] bcm2835_codec: module is from the staging directory, the quality is unknown, you have been warned. [ 4.434953] bcm2835-codec bcm2835-codec: Device registered as /dev/video10 [ 4.435002] bcm2835-codec bcm2835-codec: Loaded V4L2 decode [ 4.442628] bcm2835-isp bcm2835-isp: Device node output[0] registered as /dev/video13 [ 4.443246] bcm2835-isp bcm2835-isp: Device node capture[0] registered as /dev/video14 [ 4.443830] bcm2835-isp bcm2835-isp: Device node capture[1] registered as /dev/video15 [ 4.444044] bcm2835-codec bcm2835-codec: Device registered as /dev/video11 [ 4.444197] bcm2835-codec bcm2835-codec: Loaded V4L2 encode [ 4.446794] bcm2835-isp bcm2835-isp: Device node stats[2] registered as /dev/video16 [ 4.446830] bcm2835-isp bcm2835-isp: Register output node 0 with media controller [ 4.446870] bcm2835-isp bcm2835-isp: Register capture node 1 with media controller [ 4.446893] bcm2835-isp bcm2835-isp: Register capture node 2 with media controller [ 4.446915] bcm2835-isp bcm2835-isp: Register capture node 3 with media controller [ 4.449570] bcm2835-isp bcm2835-isp: Loaded V4L2 bcm2835-isp [ 4.464629] bcm2835-codec bcm2835-codec: Device registered as /dev/video12 [ 4.464680] bcm2835-codec bcm2835-codec: Loaded V4L2 isp [ 4.578817] cfg80211: Loading compiled-in X.509 certificates for regulatory database [ 4.661737] [drm] Initialized v3d 1.0.0 20180419 for fec00000.v3d on minor 1 [ 4.703824] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' [ 4.804584] brcmfmac: F1 signature read @0x18000000=0x15264345 [ 4.815804] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [ 4.816927] usbcore: registered new interface driver brcmfmac [ 4.835968] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt failed with error -2 [ 5.080381] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [ 5.091749] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Jan 4 2021 19:56:29 version 7.45.229 (617f1f5 CY) FWID 01-2dbd9d2e [ 5.396601] enc28j60 spi0.0: Ethernet driver 1.02 loaded [ 6.556777] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 6.614403] br0: port 1(eth0) entered blocking state [ 6.614425] br0: port 1(eth0) entered disabled state [ 6.614808] device eth0 entered promiscuous mode [ 6.629583] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay) [ 6.630108] bcmgenet fd580000.ethernet eth0: Link is Down [ 6.639019] br0: port 2(eth1) entered blocking state [ 6.639040] br0: port 2(eth1) entered disabled state [ 6.639367] device eth1 entered promiscuous mode [ 6.681467] br0: port 3(eth2) entered blocking state [ 6.681489] br0: port 3(eth2) entered disabled state [ 6.681805] device eth2 entered promiscuous mode [ 6.698015] enc28j60 spi0.0 eth2: link down [ 6.698045] enc28j60 spi0.0 eth2: promiscuous mode [ 6.698353] enc28j60 spi0.0 eth2: promiscuous mode [ 6.726965] br0: port 3(eth2) entered blocking state [ 6.726990] br0: port 3(eth2) entered forwarding state [ 6.727073] br0: port 2(eth1) entered blocking state [ 6.727111] br0: port 2(eth1) entered forwarding state [ 6.839727] uart-pl011 fe201000.serial: no DMA platform data [ 6.984953] random: crng init done [ 6.984976] random: 7 urandom warning(s) missed due to ratelimiting [ 7.069481] 8021q: 802.1Q VLAN Support v1.8 [ 7.284233] Adding 102396k swap on /var/swap. Priority:-2 extents:1 across:102396k SSFS [ 7.615972] enc28j60 spi0.0 eth2: promiscuous mode [ 7.684276] br0: port 2(eth1) entered disabled state [ 7.684646] br0: port 3(eth2) entered disabled state [ 10.794502] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off [ 10.794567] br0: port 1(eth0) entered blocking state [ 10.794586] br0: port 1(eth0) entered forwarding state [ 10.794831] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready [ 13.751174] Bluetooth: Core ver 2.22 [ 13.751288] NET: Registered protocol family 31 [ 13.751304] Bluetooth: HCI device and connection manager initialized [ 13.751335] Bluetooth: HCI socket layer initialized [ 13.751358] Bluetooth: L2CAP socket layer initialized [ 13.751392] Bluetooth: SCO socket layer initialized [ 13.760931] Bluetooth: HCI UART driver ver 2.3 [ 13.760954] Bluetooth: HCI UART protocol H4 registered [ 13.761074] Bluetooth: HCI UART protocol Three-wire (H5) registered [ 13.761389] Bluetooth: HCI UART protocol Broadcom registered [ 13.938493] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 [ 13.938505] Bluetooth: BNEP filters: protocol multicast [ 13.938520] Bluetooth: BNEP socket layer initialized [ 18.154199] Bluetooth: hci0: command 0x1003 tx timeout [ 20.234203] Bluetooth: hci0: command 0x1001 tx timeout [ 22.314204] Bluetooth: hci0: command 0x1009 tx timeout

AndysRobotics commented 3 years ago

Further testing

Pinging from my PC (Windows) with wireshark has revealed that the bridge/SPI Ethernet is definitely adding 4 additional bytes to the payload.

Image has 3 wireshark traces. The client is moved from the SPI to the USB and the ping responses are different. When pinging the host or the client in the USB ethernet port the ping response is 60 bytes (14 payload + ICMP/network frames) and ends with 4 lots of 0x00 0x00 0x00 0x00. However when the client is connected to the SPI ethernet port, there is 4 random bytes (different on every single ping response) appended to the end.

Also, if the network is connected to the SPI ethernet port, you can ping the host or client and not get the additional bytes of data.

Ping PC (10.0.1.20)->Network->SPIEth->Br0->Host (10.0.1.106) No extra bytes PC (10.0.1.20)->Network->SPIEth->Br0->Client (10.0.1.135) No extra bytes PC (10.0.1.20)->Network->PiGigEth->Br0->Host (10.0.1.106) No extra bytes PC (10.0.1.20)->Network->PiGigEth->Br0->USBEth->Client (10.0.1.135) No extra bytes PC (10.0.1.20)->Network->PiGigEth->Br0->SPIEth->Client (10.0.1.135) 4 extra bytes (random)

The 4 extra bytes only seem to occur when traffic crosses the bridge from the SPIEth indicating a bug in the bridge, SPI driver or interaction between the 2.

These 4 bytes do not match any hardware or IP addresses and appear to be completely random between pings. I have concerns these could be memory overflow and could potentially contain part of sensitive data.

image

AndysRobotics commented 3 years ago

Wireshark shows the 4 bytes as Eth.FCS (frame check sequence - 4 byte CRC)

I have compiled the drive for the ENC28J60 and disabled the FCS on the physical chip - TXCRCEN and CRCEN flags in MACON3 and ERXFCON registers as per the ENC28J60 datasheet.

The behavior does not change with this change.

I have found this email archive with a similar issue (between WLAN and ETH) https://lists.linuxfoundation.org/pipermail/bridge/2004-August/004159.html

When you ping the client from my PC with a packet size of 1468 the total size of the response is 1514. This is still 4 bytes short of the max Eth frame size of 1518 so the checksum being present is not causing the bug, if anything it looks like the fact that the checksum is not present on the other Ethernet ports is the only reason they are working.

I am not sure if this is an issue with the way the ethernet driver presents itself to the kernel/bridge or with the bridge itself. I have no idea where to take this bug now?

pelwell commented 3 years ago

I don't think this is a Pi-specific problem. You might have more luck emailing the Bridge list where you found the archived issue.

dArKsTr666 commented 3 years ago

#