search

dentproject / dentOS

dentOS SwitchDev based NOS
Other
207 stars 60 forks source link

All inbound traffic is dropped #228

Closed paulmenzel closed 3 months ago

paulmenzel commented 1 year ago

Using an Edgecore AS5114-48X we noticed all traffic to the only connected port is dropped since two days (rx_dropped:24236585). This is confirmed by using tcpdump. The management port still works.

root@ec-as5114-48x-03:~# uptime
 16:26:45 up 34 days,  6:06,  2 users,  load average: 1.83, 1.69, 1.84
root@ec-as5114-48x-03:~# uname -a
Linux ec-as5114-48x-03.molgen.mpg.de 5.10.4 #1 SMP PREEMPT Thu Apr 8 17:21:22 UTC 2021 aarch64 GNU/Linux
root@ec-as5114-48x-03:~# more /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@ec-as5114-48x-03:~# more /etc/issue
DENT OS DENTOS-dentos, 2021-04-08.17:00-da2abb3
root@ec-as5114-48x-03:~# cd /sys/class/net/swp48/statistics
root@ec-as5114-48x-03:/sys/class/net/swp48/statistics# grep . *
collisions:0
multicast:51838393
rx_bytes:98891848952
rx_compressed:0
rx_crc_errors:0
rx_dropped:24236585
rx_errors:0
rx_fifo_errors:0
rx_frame_errors:0
rx_length_errors:0
rx_missed_errors:0
rx_nohandler:0
rx_over_errors:0
rx_packets:375047233
tx_aborted_errors:0
tx_bytes:34542854
tx_carrier_errors:0
tx_compressed:0
tx_dropped:0
tx_errors:0
tx_fifo_errors:0
tx_heartbeat_errors:0
tx_packets:255556
tx_window_errors:0

Is this a known issue? We haven’t restarted the system yet, if you have an idea how to debug this issue.

paulmenzel commented 1 year ago

After some more hours:

root@ec-as5114-48x-03:/sys/class/net/swp48/statistics# grep . *
collisions:0
multicast:53080526
rx_bytes:102283504693
rx_compressed:0
rx_crc_errors:0
rx_dropped:33011159
rx_errors:0
rx_fifo_errors:0
rx_frame_errors:0
rx_length_errors:0
rx_missed_errors:0
rx_nohandler:0
rx_over_errors:0
rx_packets:383821807
tx_aborted_errors:0
tx_bytes:37470208
tx_carrier_errors:0
tx_compressed:0
tx_dropped:0
tx_errors:0
tx_fifo_errors:0
tx_heartbeat_errors:0
tx_packets:294756
tx_window_errors:0

Difference to the former values:

  1. rx_dropped: 33011159 - 24236585 = 8774574
  2. rx_packets: 383821807 - 375047233 = 8774574
taraschornyiplv commented 1 year ago

Hi @paulmenzel my understanding is that all rx packers are dropped? Is the correct? Can you please try to use CPU traffic (ping, arp) and capture it with tcpdump

Also what module are you using?

And what is driver and firmware version?

paulmenzel commented 1 year ago

Thank you for your reply.

my understanding is that all rx packers are dropped? Is the correct?

Next to the working(!) management port, that works, we only have port 48 connected/populated.

Something changed today. The link went down, while I was collecting the logs.

# last | head -3
root     pts/2        172.20.200.1     Tue Jun 20 12:22 - 12:39  (00:16)
root     pts/1        172.20.200.1     Tue Jun 20 12:13   still logged in
root     pts/0        172.20.200.1     Tue Jun 20 11:48   still logged in

# dmesg -T
[…]
[Tue Jun 20 12:07:35 2023] Aldrin2 0000:01:00.0 swp48: Link is Down
[Tue Jun 20 12:07:35 2023] br0: port 48(swp48) entered disabled state

Then we went to the switch, and reconnected the cable:

[Tue Jun 20 12:32:47 2023] Aldrin2 0000:01:00.0 swp48: Link is Up - 10Gbps/Full - flow control off
[Tue Jun 20 12:32:47 2023] br0: port 48(swp48) entered blocking state
[Tue Jun 20 12:32:50 2023] Aldrin2 0000:01:00.0 swp48: Link is Down
[Tue Jun 20 12:32:50 2023] br0: port 48(swp48) entered disabled state

But as you can see, it went down after three seconds.

The following information is from before the link went down the first time.

Can you please try to use CPU traffic (ping, arp) and capture it with tcpdump

# ip route
default via 172.20.0.1 dev ma1 proto kernel onlink 
141.14.16.0/20 dev br0 proto kernel scope link src 141.14.16.75 offload linkdown 
172.20.0.0/16 dev ma1 proto kernel scope link src 172.20.4.6
# ip neigh
172.20.200.1 dev ma1 lladdr 64:00:6a:61:23:74 DELAY
141.14.16.215 dev br0  FAILED
172.20.0.2 dev ma1  FAILED
172.20.0.1 dev ma1 lladdr 40:a6:b7:b6:d6:cd STALE

With the link down, we are unable to collect any output. But yesterday, we saw ARP requests going out, but not coming back, and confirmed that on the “main router”. We used the command below on the switch:

# tcpdump -e -i swp48 -n

Also what module are you using?

How can I find out? Is it in the output of onlpdump linked at the end?

And what is driver and firmware version?

# ethtool -i swp48
driver: prestera_sw
version: 5.10.4
firmware-version: 2.8.0
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
# modinfo prestera
filename:       /lib/modules/5.10.4/kernel/drivers/net/ethernet/marvell/prestera/prestera.ko
version:        2.0.0
description:    Marvell Prestera switch driver
license:        GPL
author:         Marvell Semi.
srcversion:     E269C3027842A9E16F6A096
depends:        
intree:         Y
name:           prestera
vermagic:       5.10.4 SMP preempt mod_unload modversions aarch64
parm:           trap_policer_profile:byte
# ls -l /lib/firmware/marvell/mvsw_prestera_fw.img
-rwxr-xr-x 1 root root 13707012 Jan 20  2021 /lib/firmware/marvell/mvsw_prestera_fw.img
# md5sum /lib/firmware/marvell/mvsw_prestera_fw.img
30033fccfeaa7c57d7a9d734817982ec  /lib/firmware/marvell/mvsw_prestera_fw.img
  1. Output of dmesg
  2. Output of onlpdump
paulmenzel commented 1 year ago

Looking at the timestamp of onlpdump, I copy-pasted the output there, it’s very likely the link went down, when running onlpdump.

paulmenzel commented 1 year ago

Today, after replugging the link, and it was up for three seconds, tcpdump captured the packets below:

# tcpdump -e -i swp48 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on swp48, link-type EN10MB (Ethernet), capture size 262144 bytes
12:33:59.707559 34:ef:b6:ec:30:34 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: fe80::36ef:b6ff:feec:3034 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:33:59.769579 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Proposal, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
12:34:00.263537 34:ef:b6:ec:30:34 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: fe80::36ef:b6ff:feec:3034 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:34:00.704990 34:ef:b6:ec:30:34 > 01:80:c2:00:00:0e, ethertype LLDP (0x88cc), length 239: LLDP, length 225: ec-as5114-48x-03.molgen.mpg.de
12:34:01.076370 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Proposal, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
351332 packets dropped by interface
taraschornyiplv commented 1 year ago

based on onlpdump i see that RX_LOS is up on port 48 Port 48: Present, Status = 0x00000004 [ RX_LOS ] eeprom:

not sure but i think it might be related to #152

can you please check your cpld version. also, I'd recommend updating to latest dentos release https://repos.refinery.dev/service/rest/repository/browse/dent/releases/org/dent/3.1/

paulmenzel commented 1 year ago

based on onlpdump i see that RX_LOS is up on port 48

Port 48: Present, Status = 0x00000004 [ RX_LOS ]
eeprom:

onlpdump still shows the same as above, but NO-CARRIER is still shown:

# ip link show dev swp48
53: swp48: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master br0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:ef:b6:ec:30:34 brd ff:ff:ff:ff:ff:ff

can you please check your cpld version.

Is it possible in Linux and not U-Boot? #152 mentions for U-Boot:

Marvell>> i2c dev 2
Marvell>> i2c md 0x40 01 1
0001: 01
Marvell>> i2c md 0x40 ff 1
00ff: 03

Any idea, what i2cget/i2cdump commands I need to run. The one below does not work:

# ls -l /sys/bus/i2c/drivers/as4224_cpld
total 0
lrwxrwxrwx 1 root root    0 Jun 20 16:53 0-0040 -> ../../../../devices/platform/ap806/ap806:config-space@f0000000/f0511000.i2c/i2c-0/0-0040
--w------- 1 root root 4096 Jun 20 16:53 bind
lrwxrwxrwx 1 root root    0 Jun 20 16:53 module -> ../../../../module/arm64_accton_as4224_cpld
--w------- 1 root root 4096 May 16 10:21 uevent
--w------- 1 root root 4096 Jun 20 16:53 unbind
# i2cget -y 2 0x40
Error: Read failed
# i2cdump -y 2 0x40
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
50: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
60: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
70: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
80: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
90: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
a0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
b0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
paulmenzel commented 1 year ago

Just some more data following the logs captured in issue https://github.com/dentproject/dentOS/issues/152:

# dmesg | grep sfp | grep module
[   24.330208] sfp sfp-47: module BROCADE          57-0000075-01    rev A    sn AAA212203415662  dc 120516  
[   24.490221] sfp sfp-48: module OPNEXT,INC       TRS5020EN-S301   rev 0000 sn 0852A-81046      dc 08122600
[   24.746207] sfp sfp-5: module Foundry Networks GBIC-SFP+SR-B    rev B4   sn M4093012         dc 151125  
[   24.910228] sfp sfp-6: module BROCADE          57-0000075-01    rev A    sn AAA212473613732  dc 121127  
[   25.043835] sfp sfp-7: module Foundry Networks GBIC-SFP+SR-B    rev B4   sn M4064742         dc 141209  
[   25.163730] sfp sfp-8: module BROCADE          57-0000075-01    rev A    sn AAA210393013231  dc 101002  
[   25.325125] sfp sfp-9: module BROCADE          57-0000075-01    rev A    sn AAF211210000K91  dc 110519
# ethtool -S swp48
NIC statistics:
     good_octets_received: 102704111354
     bad_octets_received: 0
     mac_trans_error: 0
     broadcast_frames_received: 279112157
     multicast_frames_received: 53437914
     frames_64_octets: 86496604
     frames_65_to_127_octets: 211357899
     frames_128_to_255_octets: 17291358
     frames_256_to_511_octets: 25333796
     frames_512_to_1023_octets: 1640254
     frames_1024_to_max_octets: 44125435
     excessive_collision: 0
     multicast_frames_sent: 248186
     broadcast_frames_sent: 7744
     fc_sent: 0
     fc_received: 0
     buffer_overrun: 35139252
     undersize: 0
     fragments: 0
     oversize: 0
     jabber: 0
     rx_error_frame_received: 0
     bad_crc: 0
     collisions: 0
     late_collision: 0
     unicast_frames_received: 53399829
     unicast_frames_sent: 39516
     sent_multiple: 0
     sent_deferred: 0
     good_octets_sent: 37521528
# more /sys/class/net/swp48/operstate
down
# ethtool -m swp48
    Identifier                                : 0x03 (SFP)
    Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
    Connector                                 : 0x07 (LC)
    Transceiver codes                         : 0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
    Transceiver type                          : 10G Ethernet: 10G Base-LR
    Encoding                                  : 0x06 (64B/66B)
    BR, Nominal                               : 10300MBd
    Rate identifier                           : 0x00 (unspecified)
    Length (SMF,km)                           : 10km
    Length (SMF)                              : 10000m
    Length (50um)                             : 0m
    Length (62.5um)                           : 0m
    Length (Copper)                           : 0m
    Length (OM3)                              : 0m
    Laser wavelength                          : 1310nm
    Vendor name                               : OPNEXT,INC
    Vendor OUI                                : 00:0b:40
    Vendor PN                                 : TRS5020EN-S301
    Vendor rev                                : 0000
    Option values                             : 0x00 0x1a
    Option                                    : RX_LOS implemented
    Option                                    : TX_FAULT implemented
    Option                                    : TX_DISABLE implemented
    BR margin, max                            : 0%
    BR margin, min                            : 0%
    Vendor SN                                 : 0852A-81046
    Date code                                 : 08122600
    Optical diagnostics support               : Yes
    Laser bias current                        : 33.388 mA
    Laser output power                        : 0.7304 mW / -1.36 dBm
    Receiver signal average optical power     : 0.4567 mW / -3.40 dBm
    Module temperature                        : 26.73 degrees C / 80.11 degrees F
    Module voltage                            : 3.3705 V
    Alarm/warning flags implemented           : Yes
    Laser bias current high alarm             : Off
    Laser bias current low alarm              : Off
    Laser bias current high warning           : Off
    Laser bias current low warning            : Off
    Laser output power high alarm             : Off
    Laser output power low alarm              : Off
    Laser output power high warning           : Off
    Laser output power low warning            : Off
    Module temperature high alarm             : Off
    Module temperature low alarm              : Off
    Module temperature high warning           : Off
    Module temperature low warning            : Off
    Module voltage high alarm                 : Off
    Module voltage low alarm                  : Off
    Module voltage high warning               : Off
    Module voltage low warning                : Off
    Laser rx power high alarm                 : Off
    Laser rx power low alarm                  : Off
    Laser rx power high warning               : Off
    Laser rx power low warning                : Off
    Laser bias current high alarm threshold   : 69.852 mA
    Laser bias current low alarm threshold    : 15.508 mA
    Laser bias current high warning threshold : 69.852 mA
    Laser bias current low warning threshold  : 15.508 mA
    Laser output power high alarm threshold   : 2.2387 mW / 3.50 dBm
    Laser output power low alarm threshold    : 0.0603 mW / -12.20 dBm
    Laser output power high warning threshold : 1.1220 mW / 0.50 dBm
    Laser output power low warning threshold  : 0.1514 mW / -8.20 dBm
    Module temperature high alarm threshold   : 75.00 degrees C / 167.00 degrees F
    Module temperature low alarm threshold    : -5.00 degrees C / 23.00 degrees F
    Module temperature high warning threshold : 70.00 degrees C / 158.00 degrees F
    Module temperature low warning threshold  : 0.00 degrees C / 32.00 degrees F
    Module voltage high alarm threshold       : 3.6300 V
    Module voltage low alarm threshold        : 2.9700 V
    Module voltage high warning threshold     : 3.4650 V
    Module voltage low warning threshold      : 3.1350 V
    Laser rx power high alarm threshold       : 2.2387 mW / 3.50 dBm
    Laser rx power low alarm threshold        : 0.0145 mW / -18.39 dBm
    Laser rx power high warning threshold     : 1.1220 mW / 0.50 dBm
    Laser rx power low warning threshold      : 0.0363 mW / -14.40 dBm
paulmenzel commented 1 year ago

It looks like it’s I²C bus 0 in dentOS:

# ls -l /sys/bus/i2c/drivers/as4224_cpld/
total 0
lrwxrwxrwx 1 root root    0 Jun 20 16:53 0-0040 -> ../../../../devices/platform/ap806/ap806:config-space@f0000000/f0511000.i2c/i2c-0/0-0040
--w------- 1 root root 4096 Jun 20 16:53 bind
lrwxrwxrwx 1 root root    0 Jun 20 16:53 module -> ../../../../module/arm64_accton_as4224_cpld
--w------- 1 root root 4096 May 16 10:21 uevent
--w------- 1 root root 4096 Jun 20 16:53 unbind

On bus 0 and address 0x40 I get:

# i2cdump -f -y 0 0x40
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 80 01 ff 07 0f cc cc cc cc cc cc cc cc cc cc cc    ??.?????????????
10: ff 03 3f cc 01 cc cc cc cc cc cc cc cc cc cc cc    .???????????????
20: ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc    .???????????????
30: ff ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc    ....????????????
40: cc cc cc 0e cc cc cc cc cc cc cc cc cc cc cc cc    ????????????????
50: 0d 4a 03 00 7f cc cc cc cc cc cc cc cc cc cc cc    ?J?.????????????
60: 01 71 1e cc cc cc cc cc cc cc cc cc cc cc cc cc    ?q??????????????
70: 7f 7f 7f 7f 7f cc cc cc cc cc cc cc cc cc cc cc    ????????????????
80: 6c 69 69 69 68 cc cc cc cc cc cc cc cc cc cc cc    liiih???????????
90: 02 00 71 71 cc cc cc cc cc cc cc cc cc cc cc cc    ?.qq????????????
a0: ff ff ff ff ff ff ff ff ff ff ff 7f cc cc cc cc    ...........?????
b0: ff ff ff ff ff ff 00 00 00 00 00 00 cc cc cc cc    ............????
c0: 0f fe ff ff ff 3f 00 00 00 00 00 00 cc cc cc cc    ??...?......????
d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc    ????????????????
e0: 00 71 cc cc cc cc cc cc cc cc cc cc cc cc cc cc    .q??????????????
f0: 41 53 35 31 31 34 00 00 00 00 00 00 41 57 53 05    AS5114......AWS?

If I am not mistaken, 0x40 01 1 and 0x40 ff 1 would result in 01 and 05, so the device has CPLD firmware 1.05.

paulmenzel commented 1 year ago

[This is still with CPLD firmware 1.05.]

This morning, the interface stopped working again. Some traces for the record:

Despite using interface swp48 packets from the management interface show up:

@ec-as5114-48x-03:~# tcpdump -e -i swp48 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on swp48, link-type EN10MB (Ethernet), capture size 262144 bytes
07:15:40.563126 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:42.563231 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:44.563211 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:46.563325 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:48.563693 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:50.008776 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 0, length 64
07:15:50.563132 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:51.009757 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 1, length 64
07:15:52.010747 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 2, length 64
07:15:52.563225 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:53.011910 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 3, length 64
07:15:54.012905 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 4, length 64
07:15:54.563496 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:55.013923 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 5, length 64
07:15:55.056908 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype ARP (0x0806), length 42: Request who-has 141.14.21.78 tell 141.14.16.75, length 28
07:15:56.014935 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 6, length 64
07:15:56.080905 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype ARP (0x0806), length 42: Request who-has 141.14.21.78 tell 141.14.16.75, length 28
07:15:56.563597 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:15:57.016003 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype IPv4 (0x0800), length 98: 172.20.4.6 > 141.14.21.78: ICMP echo reply, id 34081, seq 7, length 64
07:15:57.104911 34:ef:b6:ec:30:05 > f0:1f:af:e1:5b:a2, ethertype ARP (0x0806), length 42: Request who-has 141.14.21.78 tell 141.14.16.75, length 28
07:15:58.563730 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:16:00.563042 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36

ping from outside and inside does not work:

@ec-as5114-48x-03:~# tcpdump -e -i swp48 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on swp48, link-type EN10MB (Ethernet), capture size 262144 bytes
07:18:54.562943 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:18:56.563144 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:18:58.184980 34:ef:b6:ec:30:05 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 141.14.31.7 tell 141.14.16.75, length 28
07:18:58.562968 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:18:59.216929 34:ef:b6:ec:30:05 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 141.14.31.7 tell 141.14.16.75, length 28
07:19:00.240939 34:ef:b6:ec:30:05 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 141.14.31.7 tell 141.14.16.75, length 28
07:19:00.562945 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:02.563053 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:04.563002 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:06.563172 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:08.033923 34:ef:b6:ec:30:05 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 141.14.21.78 tell 172.20.4.6, length 28
07:19:08.563521 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:08.733851 34:ef:b6:ec:30:34 > 01:80:c2:00:00:0e, ethertype LLDP (0x88cc), length 239: LLDP, length 225: ec-as5114-48x-03.molgen.mpg.de
07:19:09.040917 34:ef:b6:ec:30:05 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 141.14.21.78 tell 172.20.4.6, length 28
07:19:10.068923 34:ef:b6:ec:30:05 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 141.14.21.78 tell 172.20.4.6, length 28
07:19:10.562951 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:12.563032 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:14.563301 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:16.563417 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:18.563511 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:20.562934 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:22.562938 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:24.563566 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:26.563745 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
07:19:28.563767 34:ef:b6:ec:30:34 > 01:80:c2:00:00:00, 802.3, length 39: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.34:ef:b6:ec:30:05.8030, length 36
^C
25 packets captured
25 packets received by filter
0 packets dropped by kernel
3540 packets dropped by interface
paulmenzel commented 1 year ago

The CPLD firmware is now at version 1.09. Also the latest U-Boot/ONIE environment is now installed:

-U-Boot 2019.10-10.0.0-gef8ade1 (Nov 06 2020 - 17:47:36 +0800) - 10.3.3.0.1
+U-Boot 2019.10-10.0.0-gc15e598 (Nov 30 2021 - 16:43:48 +0800) - 10.3.3.0.3

For posterity the logs from the firmware upgrade. Configure switch to reboot into ONIE rescue mode (U-Boot), then mount the data partition, copy the update file, and run it:

Loading Open Network Install Environment ...
Platform: arm64-accton_as4224-r0
Version : 2020.02.00.09
SF: Detected mx25l12845g with page size 256 Bytes, erase size 4 KiB, total 16 MiB
device 0 offset 0x210000, size 0xdf0000
SF: 14614528 bytes @ 0x210000 Read: OK
## Loading kernel from FIT Image at 20000000 ...
   Using 'accton_as4224' configuration
   Trying 'kernel' kernel subimage
     Description:  accton_as4224-r0 arm64 Kernel
     Created:      2020-11-06  10:34:04 UTC
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x200000e4
     Data Size:    7044020 Bytes = 6.7 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x80080000
     Entry Point:  0x80080000
     Hash algo:    crc32
     Hash value:   337cfb15
   Verifying Hash Integrity ... crc32+ OK
## Loading ramdisk from FIT Image at 20000000 ...
   Using 'accton_as4224' configuration
   Trying 'initramfs' ramdisk subimage
     Description:  initramfs
     Created:      2020-11-06  10:34:04 UTC
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0x206b7d74
     Data Size:    2887748 Bytes = 2.8 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x00000000
     Entry Point:  0x00000000
     Hash algo:    crc32
     Hash value:   fac8725e
   Verifying Hash Integrity ... crc32+ OK
## Loading fdt from FIT Image at 20000000 ...
   Using 'accton_as4224' configuration
   Trying 'dtb' fdt subimage
     Description:  accton_as4224-r0.dtb
     Created:      2020-11-06  10:34:04 UTC
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x20978e98
     Data Size:    27982 Bytes = 27.3 KiB
     Architecture: AArch64
     Load Address: 0x01000000
     Hash algo:    crc32
     Hash value:   e6709a05
   Verifying Hash Integrity ... crc32+ OK
   Loading fdt from 0x20978e98 to 0x01000000
   Booting using the fdt blob at 0x1000000
   Uncompressing Kernel Image
   Loading Ramdisk to 03d3e000, end 03fff044 ... OK
   Using Device Tree in place at 0000000001000000, end 0000000001009d4d

Starting kernel ...

[    2.059202] No board info available!
[    2.220237] CPUidle arm: Failed to register cpuidle driver
Info: Mounting kernel filesystems... done.
Info: Using eth0 MAC address: 34:ef:b6:ec:30:e8
Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: 172.20.4.6 / 255.255.0.0
Starting: klogd... done.
Starting: dropbear ssh daemon... done.
Starting: telnetd... done.
discover: Rescue mode detected.  Installer disabled.

Please press Enter to activate this console.
To check the install status inspect /var/log/onie.log.
Try this:  tail -f /var/log/onie.log

** Rescue Mode Enabled **
ONIE:/ # mount LABEL="ONL-DATA" /mnt/usb
ONIE:~ # cp -a /mnt/usb/root/as5114-bptfr_v01.01.09_as4224-bptfr_v0c.02.0b.updater .
ONIE:~ # umount /mnt/usb
ONIE:~ # install_url as5114-bptfr_v01.01.09_as4224-bptfr_v0c.02.0b.updater 
NOTICE: The 'install_url' command is deprecated and will be removed in 2019.02.
NOTICE: Use 'onie-nos-install' instead.
Verifying image checksum ... OK.
Preparing image archive ... OK.
Firmware: Architecture  : arm64
Firmware: Machine       : accton_as4224
Firmware: Machine Rev   : 0
Firmware: Type          : CPLD
Firmware: Version       : 11.30.18.04
Firmware: Build Date    : 2021-11-30T18:04+08:00
Updater : Version       : Fri May 14 15:27:53 2021 +0800
=== MAIN_CPLD as5114_cpld1-bptfr_v01.01.09h_20210610_2100.vme ===

65 09 4B 5F 5F 5F 5F 31 32 2E 31 F1 63 47 44 69 
61 6D 6F 6E 64 20 44 65 70 6C 6F 79 6D 65 6E 74 
20 54 6F 6F 6C 20 33 2E 31 31 0A 43 52 45 41 54 
Update CPLD code from memory buffer 0xffffafd68010, filesize=0xC8696
Product ID from CPLD: 0x80
-->
Diamond Deployment Tool 3.11
CREATION DATE: Thu Jun 10 21:45:44 2021

Expected CRC:   0x094B
Calculated CRC: 0x094B
Return: 0x00000000
<--
Product ID from CPLD: 0x88
Wait 10 seconds to release hitless
Release hitless
Function returned success
exec_result: 0
clock_counts: 3687247
spend_time=107s
ONIE:~ # Connection to ec-as5114-48x-03 closed by remote host.
Connection to ec-as5114-48x-03 closed.
# i2cdump -f -y 0 0x40
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 88 01 ff 07 0f cc cc cc cc cc cc cc cc cc cc cc    ??.?????????????
10: ff 03 3f cc 01 cc cc cc cc cc cc cc cc cc cc cc    .???????????????
20: ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc    .???????????????
30: 03 00 00 00 cc cc cc cc cc cc cc cc cc cc cc cc    ?...????????????
40: cc cc cc 0e cc cc cc cc cc cc cc cc cc cc cc cc    ????????????????
50: 0d 4a 03 00 7f cc cc cc cc cc cc cc cc cc cc cc    ?J?.????????????
60: 01 71 1e cc cc cc cc cc cc cc cc cc cc cc cc cc    ?q??????????????
70: 7f 7f 7f 7f 7f cc cc cc cc cc cc cc cc cc cc cc    ????????????????
80: 6a 69 69 69 68 cc cc cc cc cc cc cc cc cc cc cc    jiiih???????????
90: 02 00 71 71 cc cc cc cc cc cc cc cc cc cc cc cc    ?.qq????????????
a0: 00 00 00 00 00 00 f0 01 00 00 00 c0 cc cc cc cc    ......??...?????
b0: 00 00 00 00 00 00 00 00 00 00 00 00 cc cc cc cc    ............????
c0: 0f fe ff ff ff 3f 00 00 00 00 00 00 cc cc cc cc    ??...?......????
d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc    ????????????????
e0: 00 71 cc cc cc cc cc cc cc cc cc cc cc cc cc 00    .q?????????????.
f0: 41 53 35 31 31 34 00 00 00 00 00 00 41 57 53 09    AS5114......AWS?
taraschornyiplv commented 1 year ago

Hi @paulmenzel. Did CPLD firmware update resolve this issue?

paulmenzel commented 3 months ago

Please excuse the lack of response. To my knowledge the problem was fixed. Thank you!