Memory and mavlink problems pixhawk v2

tuloski commented 5 years ago

I noticed that my pixhawk v2 was having some problems, so I hooked up the console in telem 4/5.

What I see at random are different problems:

Memory problems: this warning comes out "WARNING WARNING WARNING! Revision A has a silicon errata: This device can only utilize a maximum of 1MB flash safely! https://docs.px4.io/en/flight_controller/silicon_errata.html" but I havn't found any way to fix it. And it's not only a warning because later this happens:

"nsh: mount: mount failed: Out of memory nsh: mkfatfs: mkfatfs failed: Out of memory"

Mavlink problems: I have a custom "extras" script to launch mavlink instances. In particular the script is the following:
```
mavlink stop-all
#ESP
#mavlink start -r 20000 -b 921600 -d /dev/ttyS0
#USB
mavlink start -d /dev/ttyACM0 -b 1000000 -m custom
mavlink stream -d /dev/ttyACM0 -s HEARTBEAT -r 2
mavlink stream -d /dev/ttyACM0 -s ATTITUDE_QUATERNION -r 50
mavlink stream -d /dev/ttyACM0 -s ATTITUDE -r 10
mavlink stream -d /dev/ttyACM0 -s HIGHRES_IMU -r 50
mavlink stream -d /dev/ttyACM0 -s GPS_RAW_INT -r 5
mavlink stream -d /dev/ttyACM0 -s GLOBAL_POSITION_INT -r 50
```
what happens sometimes at random is this error: "INFO [mavlink] all instances stopped INFO [mavlink] mode: Custom, data rate: 50000 B/s on /dev/ttyACM0 @ 1000000B ERROR [mavlink] could not open /dev/ttyACM0 WARN [mavlink] mavlink for device /dev/ttyACM0 is not running WARN [mavlink] mavlink for device /dev/ttyACM0 is not running WARN [mavlink] mavlink for device /dev/ttyACM0 is not running WARN [mavlink] mavlink for device /dev/ttyACM0 is not running WARN [mavlink] mavlink for device /dev/ttyACM0 is not running WARN [mavlink] mavlink for device /dev/ttyACM0 is not running "

I really don't know where to start fixing it. Can it be all related to memory issues? Why isn't it able to open /dev/ttyACM0 sometimes?

PS: I tried to update the bootloader with https://docs.px4.io/v1.9.0/en/advanced_config/parameter_reference.html#SYS_BL_UPDATE without any success.

mhkabir commented 5 years ago

You're running out of RAM. You need to either stop some modules (unused mavlink instances for example), or get newer hardware with more RAM (e.g Pixhawk 4)

The Silicon Errata warning is about Flash, and is unrelated to the issues you're seeing.

tuloski commented 5 years ago

But it's standard px4 1.9.2 code. There should be a warning that it is unusable on pixhawk v2.

tuloski commented 5 years ago

BTW "free" returns:

nsh> free total used free largest Umem: 238064 170320 67744 64576

It seems there is some free RAM.

mhkabir commented 5 years ago

Can you post logs from before and after your extras script runs?

@dagar FYI

tuloski commented 5 years ago

Case 1 (happening at random, can't open ACM0):

FMUv2 ver 0x8 : Rev 0 V2
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V2
HW type: V2
HW version: 0x00090008
HW revision: 0x00000000
FW git-hash: 106905871d374c125730fe675266d67c7f869469
FW version: Release 1.9.2 (17367807)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 423371c7d4012e725ac4ca51323a18df64e581b3
Build datetime: Jul  9 2019 13:47:06
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255                          204]
PX4GUID: 000100000000313235333432470a00280019
MCU: STM32F42x, rev. A

WARNING   WARNING   WARNING!
Revision A has a silicon errata:
This device can only utilize a maximum of 1MB flash safely!
https://docs.px4.io/en/flight_controller/silicon_errata.html

[hardfault_log] Fault Log is Armed
INFO  [tune_control] Publishing standard tune 1
INFO  [param] selected parameter default file /fs/mtd_params
Board defaults: /etc/init.d/rc.board_defaults
rgbled on I2C bus 1 at 0x55 (bus: 100 KHz, max: 100 KHz)
nsh: rgbled_ncp5623c: command not found
nsh: rgbled_pwm: command not found
nsh: blinkm: command not found
Board sensors: /etc/init.d/rc.board_sensors
HMC5883_I2C on I2C bus 1 at 0x1e (bus: 100 KHz, max: 400 KHz)
nsh: lis3mdl: command not found
nsh: ist8310: command not found
WARN  [hmc5883] no device on bus 2 (type: 1)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 4 (1000 KHz)
WARN  [mpu6000] no device on bus #5 (SPI4)
INFO  [mpu9250] Bus probed: 3
WARN  [mpu9250] probe failed! 104
WARN  [mpu9250] no device on bus 3
INFO  [mpu9250] Bus probed: 5
WARN  [mpu9250] probe failed! 0
WARN  [mpu9250] no device on bus 5
L3GD20 on SPI bus 1 at 1 (11000 KHz)
LSM303D on SPI bus 1 at 2 (11000 KHz)
MS5611_SPI on SPI bus 4 at 3 (20000 KHz)
WARN  [ms5611] no device on bus 4
MS5611_SPI on SPI bus 1 at 3 (20000 KHz)
nsh: bst: command not found
ERROR [param] Parameter SENS_EN_BATT not found
ERROR [param] Parameter SENS_EN_MB12XX not found
ERROR [param] Parameter SENS_EN_PGA460 not found
ERROR [param] Parameter SENS_EN_SF1XX not found
ERROR [param] Parameter SENS_EN_TRANGER not found
ERROR [param] Parameter TRIG_MODE not found
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
INFO  [px4io] default PWM output device
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [init] Mixer: /etc/mixers/pass.aux.mix on /dev/pwm_output1
ERROR [param] Parameter CAM_CAP_FBACK not found
ERROR [param] Parameter WEST_EN not found
px4flow [370:100]
INFO  [px4flow] scanning I2C buses for device..
Addons script: /fs/microsd/etc/extras.txt
INFO  [mavlink] waiting for instances to stop
.INFO  [mavlink] exiting channel 0

INFO  [mavlink] all instances stopped
INFO  [mavlink] mode: Custom, data rate: 50000 B/s on /dev/ttyACM0 @ 1000000B
ERROR [mavlink] could not open /dev/ttyACM0
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [logger] logger started (mode=all)

NuttShell (NSH)

Case 2 (happening at random, out of memory):

FMUv2 ver 0x8 : Rev 0 V2
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V2
HW type: V2
HW version: 0x00090008
HW revision: 0x00000000
FW git-hash: 106905871d374c125730fe675266d67c7f869469
FW version: Release 1.9.2 (17367807)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 423371c7d4012e725ac4ca51323a18df64e581b3
Build datetime: Jul  9 2019 13:47:06
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
PX4GUID: 000100000000313235333432470a00280019
MCU: STM32F42x, rev. A

WARNING   WARNING   WARNING!
Revision A has a silicon errata:
This device can only utilize a maximum of 1MB flash safely!
https://docs.px4.io/en/flight_controller/silicon_errata.html

nsh: mount: mount failed: Out of memory
INFO  [tune_control] Publishing standard tune 16
nsh: mkfatfs: mkfatfs failed: Out of memory
INFO  [tune_control] Publishing standard tune 2
INFO  [param] selected parameter default file /fs/mtd_params
Board defaults: /etc/init.d/rc.board_defaults
WARN  [dataman] Could not open data manager file /fs/microsd/dataman
ERROR [dataman] dataman start failed
rgbled on I2C bus 1 at 0x55 (bus: 100 KHz, max: 100 KHz)
nsh: rgbled_ncp5623c: command not found
nsh: rgbled_pwm: command not found
nsh: blinkm: command not found
Board sensors: /etc/init.d/rc.board_sensors
HMC5883_I2C on I2C bus 1 at 0x1e (bus: 100 KHz, max: 400 KHz)
nsh: lis3mdl: command not found
nsh: ist8310: command not found
WARN  [hmc5883] no device on bus 2 (type: 1)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 4 (1000 KHz)
WARN  [mpu6000] no device on bus #5 (SPI4)
INFO  [mpu9250] Bus probed: 3
WARN  [mpu9250] probe failed! 104
WARN  [mpu9250] no device on bus 3
INFO  [mpu9250] Bus probed: 5
WARN  [mpu9250] probe failed! 0
WARN  [mpu9250] no device on bus 5
L3GD20 on SPI bus 1 at 1 (11000 KHz)
LSM303D on SPI bus 1 at 2 (11000 KHz)
MS5611_SPI on SPI bus 4 at 3 (20000 KHz)
WARN  [ms5611] no device on bus 4
MS5611_SPI on SPI bus 1 at 3 (20000 KHz)
nsh: bst: command not found
ERROR [param] Parameter SENS_EN_BATT not found
ERROR [param] Parameter SENS_EN_MB12XX not found
ERROR [param] Parameter SENS_EN_PGA460 not found
ERROR [param] Parameter SENS_EN_SF1XX not found
ERROR [param] Parameter SENS_EN_TRANGER not found
ERROR [param] Parameter TRIG_MODE not found
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
ERROR [mavlink] DM_KEY_MISSION_STATE lock failed
ERROR [mavlink] offboard mission init failed (-1)
INFO  [px4io] default PWM output device
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [init] Mixer: /etc/mixers/pass.aux.mix on /dev/pwm_output1
ERROR [param] Parameter CAM_CAP_FBACK not found
ERROR [param] Parameter WEST_EN not found
px4flow [350:100]
INFO  [px4flow] scanning I2C buses for device..
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [logger] logger started (mode=all)
INFO  [logger] log root dir created: /fs/microsd/log

NuttShell (NSH)

dagar commented 5 years ago

What's on you SD card? Can you format it and try again?

tuloski commented 5 years ago

What's on you SD card? Can you format it and try again?

There are only extras and logs. Formatted it now with no luck.

dagar commented 5 years ago

All the errors involve the sdcard. What size is it? How is it formatted? Do you have another you can try?

nsh: mount: mount failed: Out of memory
mkfatfs: mkfatfs failed: Out of memory
WARN [dataman] Could not open data manager file /fs/microsd/dataman
ERROR [dataman] dataman start failed

tuloski commented 5 years ago

All the errors involve the sdcard. What size is it? How is it formatted? Do you have another you can try?

nsh: mount: mount failed: Out of memory

mkfatfs: mkfatfs failed: Out of memory

WARN [dataman] Could not open data manager file /fs/microsd/dataman

ERROR [dataman] dataman start failed

It's a Kingstone 4Gb that ships with pixhawk. BTW I switched it with a newer one 16 Gb, and it seems solved. I'll keep you updated if it happens again. Thanks for the input.

tuloski commented 5 years ago

mmm I said hurray too early. The problem is still there. It worked 3 reboots and now again at random. I'm starting to think it's some hardware failure on the pixhawk.

julianoes commented 5 years ago

@tuloski can you see the log output when it fails?

tuloski commented 5 years ago

@tuloski can you see the log output when it fails?

In the SD?

julianoes commented 5 years ago

No, that would be with a console connected: https://dev.px4.io/en/debug/system_console.html

tuloski commented 5 years ago

Yes, actually what I wrote in Case1 and Case2 are console outputs. Also when it fails I see it.

julianoes commented 5 years ago

Oh, sorry I missed that.

tuloski commented 5 years ago

Adding some usleep of 0.5s here and there in the extras.txt seems to solve the issue. Still not clear what the real issue is.

julianoes commented 5 years ago

Adding some usleep of 0.5s here and there in the extras.txt seems to solve the issue. Still not clear what the real issue is.

That's odd though because when it fails it doesn't even mount the SD card (from what I see in the log above).

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

PX4 / PX4-Autopilot

Memory and mavlink problems pixhawk v2 #12502