search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.32k stars 13.43k forks source link

FMUV5X/6X Ethernet Not Working #20280

Closed AlexKlimaj closed 1 year ago

AlexKlimaj commented 2 years ago

On both the FMUV5X and FMUV6X with either the regular or mini Holybro carrier boards, ethernet is not working. On both carrier boards, ethernet was working for a little while, then stopped working all together.

dagar commented 2 years ago

@davids5

davids5 commented 2 years ago

This was verified on both during verification. I will re check this.

davids5 commented 2 years ago

Verified on the reference V6X rebased on main today

config
eth0    Link encap:Ethernet HWaddr 86:7e:7b:42:64:51 at UP
        inet addr:192.168.0.131 DRaddr:192.168.0.254 Mask:255.255.255.0

nsh> ping 192.168.0.10
PING 192.168.0.10 56 bytes of data
56 bytes from 192.168.0.10: icmp_seq=0 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=1 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=2 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=3 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=4 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=5 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=6 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=7 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=8 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=9 time=0 ms
10 packets transmitted, 10 received, 0% packet loss, time 10010 ms
nsh> ver akk
ERROR [ver] unknown command
nsh> ver all
HW arch: PX4_FMU_V6X
HW type: V6X000000
HW version: 0x000
HW revision: 0x000
FW git-hash: a9ed77fa5e02af32c5d194794213ccaa30999ec0
FW version: 1.13.0 80 (17629312)
FW git-branch: cuav_v6x_dev_cfg
OS: NuttX
OS version: Release 10.4.0 (168034559)
OS git-hash: 85bd35b8b906dcd87425107ff6ab2119b2c0d451
Build datetime: Sep 22 2022 06:27:42
Build uri: localhost
Build variant: default
Toolchain: GNU GCC, 9.3.1 20200408 (release)
PX4GUID: 00060000000035313536343751050023002a
MCU: STM32H7[4|5]xxx, rev. Y

Holybo V6X

nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 8a:c3:d9:f0:56:c2 at UP
        inet addr:192.168.0.245 DRaddr:192.168.0.254 Mask:255.255.255.0

nsh> INFO  [gyro_calibration] gyro 0 (2424858) updating offsets [0.004, 0.014, -0.012]->[-0.010, 0.012, 0.011] 32.5 degC

nsh>
nsh>
nsh>
nsh>
nsh> ping 192.168.0.10
PING 192.168.0.10 56 bytes of data
56 bytes from 192.168.0.10: icmp_seq=0 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=1 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=2 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=3 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=4 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=5 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=6 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=7 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=8 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=9 time=0 ms
10 packets transmitted, 10 received, 0% packet loss, time 10010 ms
nsh> ping 192.168.0.101
PING 192.168.0.101 56 bytes of data
56 bytes from 192.168.0.101: icmp_seq=0 time=187 ms
56 bytes from 192.168.0.101: icmp_seq=1 time=97 ms
56 bytes from 192.168.0.101: icmp_seq=2 time=112 ms
56 bytes from 192.168.0.101: icmp_seq=3 time=49 ms
56 bytes from 192.168.0.101: icmp_seq=4 time=72 ms
56 bytes from 192.168.0.101: icmp_seq=5 time=106 ms
56 bytes from 192.168.0.101: icmp_seq=6 time=132 ms
56 bytes from 192.168.0.101: icmp_seq=7 time=37 ms
56 bytes from 192.168.0.101: icmp_seq=8 time=3 ms
56 bytes from 192.168.0.101: icmp_seq=9 time=3 ms
10 packets transmitted, 10 received, 0% packet loss, time 10010 ms
nsh> ping 192.168.0.222
PING 192.168.0.222 56 bytes of data
ERROR: sendto failed at seqno 0: 114
nsh> ver all
HW arch: PX4_FMU_V6X
HW type: V6X000004
HW version: 0x000
HW revision: 0x004
FW git-hash: a9ed77fa5e02af32c5d194794213ccaa30999ec0
FW version: 1.13.0 80 (17629312)
FW git-branch: cuav_v6x_dev_cfg
OS: NuttX
OS version: Release 10.4.0 (168034559)
OS git-hash: 85bd35b8b906dcd87425107ff6ab2119b2c0d451
Build datetime: Sep 22 2022 06:27:42
Build uri: localhost
Build variant: default
Toolchain: GNU GCC, 9.3.1 20200408 (release)
PX4GUID: 000600000000373436373330511900460037
MCU: STM32H7[4|5]xxx, rev. V

Holybro Base + V5x

eth0    Link encap:Ethernet HWaddr ca:4f:40:2b:68:64 at UP
        inet addr:192.168.0.104 DRaddr:192.168.0.254 Mask:255.255.255.0

nsh> ping 192.168.0.10
PING 192.168.0.10 56 bytes of data
56 bytes from 192.168.0.10: icmp_seq=0 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=1 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=2 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=3 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=4 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=5 time=0 ms
56 bytes from 192.168.0.10: icmp_seq=6 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=7 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=8 time=1 ms
56 bytes from 192.168.0.10: icmp_seq=9 time=1 ms
10 packets transmitted, 10 received, 0% packet loss, time 10012 ms
nsh> ver all
HW arch: PX4_FMU_V5X
HW type: V5X000002
HW version: 0x000
HW revision: 0x002
FW git-hash: a9ed77fa5e02af32c5d194794213ccaa30999ec0
FW version: 1.13.0 80 (17629312)
FW git-branch: cuav_v6x_dev_cfg
OS: NuttX
OS version: Release 10.4.0 (168034559)
OS git-hash: 85bd35b8b906dcd87425107ff6ab2119b2c0d451
Build datetime: Sep 22 2022 11:32:43
Build uri: localhost
Build variant: default
Toolchain: GNU GCC, 9.3.1 20200408 (release)
PX4GUID: 000200000000353236353038510700430027
MCU: STM32F76xxx, rev. Z
davids5 commented 2 years ago

@AlexKlimaj - what is your test setup?

AlexKlimaj commented 2 years ago

A Holybro Regular or mini carrier board. Then a holybro fmuv5x or fmuv6x connected to a switch on the network. It worked for a while then stopped working all together. Ethernet power light on the carrier is on and sometimes the Link light will come on, but it won't talk to the switch.

davids5 commented 2 years ago

What does ifconfig Show? netman show?

AlexKlimaj commented 2 years ago

Ethernet PWR and LINK lights are solid.

[boot] Rev 0x4 : Ver 0x0 V6X000004
reset done, 10 ms
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V6X
HW type: V6X000004
HW version: 0x000
HW revision: 0x004
FW git-hash: 32277f63323a04e8ed716deaec0ebd1781c45cc8
FW version: 1.13.0 80 (17629312)
FW git-branch: main
OS: NuttX
OS version: Release 10.2.0 (167903487)
OS git-hash: 85bd35b8b906dcd87425107ff6ab2119b2c0d451
Build datetime: Sep 23 2022 13:55:29
Build uri: localhost
Build variant: default
Toolchain: GNU GCC, 9.3.1 20200408 (release)
PX4GUID: 00060000000037343637333051190036004f
MCU: STM32H7[4|5]xxx, rev. V
ault_log] Fault Log is Armed
INFO  [param] selected parameter default file /fs/mtd_params
INFO  [param] importing from '/fs/mtd_params'
WARN  [parameters] ignoring unrecognised parameter 'PWM_AUX_OUT'
WARN  [parameters] ignoring unrecognised parameter 'PWM_MAIN_OUT'
WARN  [parameters] ignoring unrecognised parameter 'SYS_CTRL_ALLOC'
INFO  [parameters] BSON document size 3108 bytes, decoded 3108 bytes (INT32:50, FLOAT:99)
INFO  [param] selected parameter backup file /fs/microsd/parameters_backup.bson
Board architecture defaults: /etc/init.d/rc.board_arch_defaults
Board defaults: /etc/init.d/rc.board_defaults
Loading airframe: /etc/init.d/airframes/4001_quad_x
INFO  [dataman] data manager file '/fs/microsd/dataman' size is 62560 bytes
INFO  [uavcan] Node ID 1, bitrate 1000000
INFO  [px4io] IO FW CRC match
Board sensors: /etc/init.d/rc.board_sensors
INFO  [ina226] Failed to init INA226 on bus 1, but will try again periodically.
ina226 #0 on I2C bus 1 (external) address 0x41
INFO  [ina226] Failed to init INA226 on bus 2, but will try again periodically.
ina226 #1 on I2C bus 2 (external) address 0x41
icm20649 #0 on SPI bus 3 rotation 6
icm42688p #0 on SPI bus 2 rotation 6
icm42670p #0 on SPI bus 1 rotation 10
bmm150 #0 on I2C bus 4 (external) address 0x10
WARN  [SPI_I2C] ist8310: no instance started (no device on bus?)
bmp388 #0 on I2C bus 4 (external) address 0x77
bmp388 #1 on I2C bus 2 (external) address 0x76
WARN  [SPI_I2C] ms5611: no instance started (no device on bus?)
nsh: battery_status: command not found
ekf2 [1012:237]
Starting Secondary GPS on /dev/ttyS7
Starting Main GPS on /dev/ttyS0
Starting MAVLink on /dev/ttyS6
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS6 @ 57600B
Starting MAVLink on ethernet
INFO  [mavlink] mode: Normal, data rate: 100000 B/s on udp port 14550 remote port 14550
INFO  [rc_input] valid device required
ERROR [rc_input] Task start failed (-1)
INFO  [logger] logger started (mode=all)

NuttShell (NSH) NuttX-10.2.0
nsh> WARN  [health_and_arming_checks] Preflight Fail: Accel 0 uncalibrated
WARN  [health_and_arming_checks] Preflight Fail: ekf2 missing data
WARN  [mavlink] no broadcasting address found
WARN  [health_and_arming_checks] Preflight Fail: Accel 0 uncalibrated
WARN  [health_and_arming_checks] Preflight Fail: Accel 2 uncalibrated
WARN  [health_and_arming_checks] Preflight Fail: Gyro 2 uncalibrated

nsh>
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 62:2e:41:07:87:52 at UP
        inet addr:0.0.0.0 DRaddr:192.168.0.254 Mask:255.255.255.0

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.0.3
ROUTER=192.168.0.254
DNS=192.168.0.254
AlexKlimaj commented 2 years ago

Okay I just received two new Holybro Mini Carrier boards. Ethernet works on one but not the other. Overall I have 4 Holybro carrier boards and ethernet only works on one. 3 mini carriers and 1 full size carrier. Ethernet only works on 1 mini carrier. @vincentpoont2

Tested with two different switches and two different cable setups.

Using the fmuv6x on the first carrier board, ethernet is not working.

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.0.3
ROUTER=192.168.0.254
DNS=192.168.0.254
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 62:2e:41:07:87:52 at UP
        inet addr:0.0.0.0 DRaddr:192.168.0.254 Mask:255.255.255.0

Using the fmuv5x on the second carrier board and ethernet is working.

NuttShell (NSH) NuttX-10.2.0
nsh> WARN  [mavlink] no broadcasting address found
INFO  [mavlink] using network interface eth0, IP: 192.168.1.50
INFO  [mavlink] with netmask: 255.255.255.0
INFO  [mavlink] and broadcast IP: 192.168.1.255
INFO  [mavlink] partner IP: 192.168.1.22

nsh>
nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.0.3
ROUTER=192.168.0.254
DNS=192.168.0.254
nsh>
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 82:57:a5:5e:cb:09 at UP
        inet addr:192.168.1.50 DRaddr:192.168.1.1 Mask:255.255.255.0

Plugging the fmuv6x into the second mini carrier, ethernet works.

WARN  [mavlink] no broadcasting address found
WARN  [health_and_arming_checks] Preflight Fail: Accel 2 uncalibrated
WARN  [health_and_arming_checks] Preflight Fail: Gyro 2 uncalibrated
INFO  [mavlink] using network interface eth0, IP: 192.168.1.56
INFO  [mavlink] with netmask: 255.255.255.0
INFO  [mavlink] and broadcast IP: 192.168.1.255
INFO  [mavlink] partner IP: 192.168.1.22

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.0.3
ROUTER=192.168.0.254
DNS=192.168.0.254
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 62:2e:41:07:87:52 at UP
        inet addr:192.168.1.56 DRaddr:192.168.1.1 Mask:255.255.255.0

Plugging the fmuv5x on the first carrier board and ethernet is not working.

Plugging an ARKV6X into the second carrier board and ethernet is working. Plugging an ARKV6X into the first carrier board and ethernet is not working.

vincentpoont2 commented 2 years ago

@AlexKlimaj Each Baseboard were test during production via connecting to the router and ping. The test method is the same as David above.

Can do check on the hardware connection between the flight controller and the baseboard, and try to secure it with the fixing screws?

davids5 commented 2 years ago

@AlexKlimaj you yave a 192.168.1 Network - and the config is for a 192.168.0 network. Edit the config 

DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.1.3 < change the 0 to 1 
ROUTER=192.168.0.254 <- List your router IP here
DNS=192.168.0.254 <- List your router IP here
AlexKlimaj commented 2 years ago

Why do I have 1 base board that works with DHCP and the rest don't work at all?

Yes I am seating the FC SOM in the baseboard securely. This looks to me like I have 3 baseboards with faulty ethernet hardware. As ethernet on all SOMs I have work on the one baseboard.

I manually set the netman config and it still doesn't work.

Regular larger carrier with FMUV5X. No ETH Link light.

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.1.50
ROUTER=192.168.1.1
DNS=192.168.1.1
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 82:57:a5:5e:cb:09 at DOWN
        inet addr:0.0.0.0 DRaddr:192.168.1.1 Mask:255.255.255.0

nsh>

Moving the same FMUV5X to the previously working Mini baseboard, and it works. ETH Link light flashing. 

INFO  [mavlink] using network interface eth0, IP: 192.168.1.50
INFO  [mavlink] with netmask: 255.255.255.0
INFO  [mavlink] and broadcast IP: 192.168.1.255

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.1.50
ROUTER=192.168.1.1
DNS=192.168.1.1
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 82:57:a5:5e:cb:09 at UP
        inet addr:192.168.1.50 DRaddr:192.168.1.1 Mask:255.255.255.0

Moving the same FMUV5X to one of the non working mini baseboards and it doesn't work. It also looks like the net.cfg on the microsd was deleted and back to the default config.

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.0.3
ROUTER=192.168.0.254
DNS=192.168.0.254
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 82:57:a5:5e:cb:09 at DOWN
        inet addr:0.0.0.0 DRaddr:192.168.0.254 Mask:255.255.255.0

Creating a new net.cfg on the sd card and rebooting. It still doesn't work.

nsh> netman show
DEVICE=eth0
BOOTPROTO=fallback
NETMASK=255.255.255.0
IPADDR=192.168.1.50
ROUTER=192.168.1.1
DNS=192.168.1.1
nsh> ifconfig
eth0    Link encap:Ethernet HWaddr 82:57:a5:5e:cb:09 at DOWN
        inet addr:0.0.0.0 DRaddr:192.168.1.1 Mask:255.255.255.0
davids5 commented 2 years ago

@AlexKlimaj - Some of the confusion might be that the net config is stored on the base in the EEPROM.

Can you scope out the circuit?

AlexKlimaj commented 2 years ago

Yeah scoping now. On the external of the connector, I have almost 50V ethernet signals. Wondering if the DC blocking caps are failing.

AlexKlimaj commented 2 years ago

Update, ethernet no longer works on the one baseboard that did have working ethernet.

davids5 commented 2 years ago

@AlexKlimaj do you want to sent me that system to trouble shoot?

AlexKlimaj commented 2 years ago

Yeah I can ship you one.

davids5 commented 2 years ago

@AlexKlimaj - the board is transmitting but not receiving. The data looks to be making it across the PAB (scoping RXD0, RXD1 and CRS_DV) to the CPU but it has errors and is not received. You could try swapping the XTAL on the Phy or the Phy.

AlexKlimaj commented 2 years ago

Hm, I wonder if the 50V from the POE switch caused the PHY failure.

davids5 commented 2 years ago

Could be. I was seeing signal at the phy but You could replace the passives and see if they were stressed and that fixes it.

image

AlexKlimaj commented 1 year ago

@vincentpoont2 I think the conclusion is my POE switch with 50V signals caused the Phy to fail.

benjinne commented 1 year ago

Hi @AlexKlimaj will the ethernet work on your flight controller? I have some of your FMUV6X FCUs, and I'm still waiting for the holybro carrier boards.

AlexKlimaj commented 1 year ago

@benjinne Yes ethernet works on my ARKV6X. This is a carrier board ethernet Phy issue.