Open JMare opened 7 years ago
JMare
commented
7 years ago Updates from debugging:
Problem seems to be in the HIL plugin, as it is reproducible with this pluginlists.yaml:
plugin_blacklist: ['*'] plugin_whitelist: ['sys_*','hil']
Have tried 0.21.0 and 0.21.2 and the problem occurs in both.
TSC21
commented
7 years ago @vooon is it possible that 13c92ad687bdc9dba96c4472bbb019d229b0556f is breaking this?
TSC21
commented
7 years ago @JMare I feel it strange that this comes from the HIL plugin. 13c92ad687bdc9dba96c4472bbb019d229b0556f only was inserted on 0.21.1. The HIL plugin does not change since 0.20.0 release, when it was inserted.
vooon
commented
7 years ago Does it work if you blacklist only hil? It feels that problem not in one single plugin, rather than somewhere in common code. And alignment is highly dependant on compiler version. Is that possible to use gcc 6.x? Also weird idea - compile in Release mode, i want to know does deb's also broken or maybe not.
If problem in to_eigen() then unfortunately we should switch to another library, problem in RVO is unacceptable.
TSC21
commented
7 years ago If problem in to_eigen() then unfortunately we should switch to another library, problem in RVO is unacceptable.
I don't think the problem is there. @JMare also tried 0.21.0 (which does not contain 13c92ad) with the same result.
JMare
commented
7 years ago It seems to work if I blacklist only HIL but I didn't run it enough times to say 100%.
On another note, I take it HIL=Hardware in the Loop? HIL could be disabled by default for APM, since ardupilot dropped support for HIL some time ago.
TSC21
commented
7 years ago On another note, I take it HIL=Hardware in the Loop? HIL could be disabled by default for APM, since ardupilot dropped support for HIL some time ago.
It could but then we will be skipping the problem and not solving it.
TSC21
commented
6 years ago @JMare can you give us some insights on this? Does it keep happening on your side? I don't have a way of reproducing it unfortunately.
JMare
commented
6 years ago @TSC21 so far I have just been running with HIL blacklisted. I will try again with latest master in the next few days and see if the problem still occurs with Raspberry Pi + APM.
TSC21
commented
6 years ago OK thank you @JMare
jannsta1
commented
6 years ago Hi, just to mention that I am setting up an MAV with raspberry pi 3 and pixhawk (using px4 flight stack) and encountered this issue. Blacklisting HIL in pluginlists.yaml also seems to prevent the segfault for my setup. Happy to provide any diagnostic info or test updated code if it helps.
esaattia
commented
6 years ago Hi,
I also am seeing seg fault on startup on the raspberry pi (Navio2) i am running ROS kinetic and was trying to run mavros 0.22 . the previos version 0.21 worked fine
TSC21
commented
6 years ago @vooon I guess the to_eigen() function is somewhat breaking this?
TSC21
commented
6 years ago @JMare @jannsta1 @esaattia can you try to revert the changes done in 13c92ad and check if that works to you?
jannsta1
commented
6 years ago @TSC21, I reverted to mavros 0.21.0 and still see the segfault (with the same traceback) with more or less the same frequency as 0.21.5. Once again blacklisting hil in px4_pluginlists.yaml more or less prevents the issue.
TSC21
commented
6 years ago @TSC21, I reverted to mavros 0.21.0 and still see the segfault (with the same traceback) with more or less the same frequency as 0.21.5. Once again blacklisting hil in px4_pluginlists.yaml more or less prevents the issue.
Ok so something else is failing there. I need to take a look at the code to check what it can be.
jannsta1
commented
6 years ago Just a note to say this has now also happened a couple of times with hil blacklisted so I would guess a race condition of some sort that is made less likely when hil is disabled.
JMare
commented
6 years ago @TSC21 can hil be blacklisted by deafult in apm_pluginlists.yaml? It won't fix this problem but there is no reason for HIL to run with APM.
I recently did another source install of mavros on Raspi3 and had the same problem with latest master, fixed by blacklisting HIL.
TSC21
commented
6 years ago @TSC21 can hil be blacklisted by deafult in apm_pluginlists.yaml? It won't fix this problem but there is no reason for HIL to run with APM.
I recently did another source install of mavros on Raspi3 and had the same problem with latest master, fixed by blacklisting HIL.
Sure, please issue a PR when you have time.
codingCoffee
commented
2 years ago Has the issue been resolved? Blacklisting hil in px4_pluginlists.yaml does not solve the issue. Any leads on how to fix the problem? Trying on RPi4
pi@raspberrypi:~/ros_catkin_ws $ roslaunch mavros apm.launch fcu_url:=/dev/ttyAMA0
... logging to /home/pi/.ros/log/cc6eafc6-bf23-11ec-9a31-dca63200a718/roslaunch-raspberrypi-24031.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://raspberrypi:45521/
SUMMARY
========
CLEAR PARAMETERS
* /mavros/
PARAMETERS
* /mavros/cmd/use_comp_id_system_control: False
* /mavros/conn/heartbeat_mav_type: ONBOARD_CONTROLLER
* /mavros/conn/heartbeat_rate: 1.0
* /mavros/conn/system_time_rate: 1.0
* /mavros/conn/timeout: 10.0
* /mavros/conn/timesync_rate: 10.0
* /mavros/distance_sensor/rangefinder_pub/field_of_view: 0.0
* /mavros/distance_sensor/rangefinder_pub/frame_id: lidar
* /mavros/distance_sensor/rangefinder_pub/id: 0
* /mavros/distance_sensor/rangefinder_pub/send_tf: False
* /mavros/distance_sensor/rangefinder_pub/sensor_position/x: 0.0
* /mavros/distance_sensor/rangefinder_pub/sensor_position/y: 0.0
* /mavros/distance_sensor/rangefinder_pub/sensor_position/z: -0.1
* /mavros/distance_sensor/rangefinder_sub/id: 1
* /mavros/distance_sensor/rangefinder_sub/orientation: PITCH_270
* /mavros/distance_sensor/rangefinder_sub/subscriber: True
* /mavros/fake_gps/eph: 2.0
* /mavros/fake_gps/epv: 2.0
* /mavros/fake_gps/fix_type: 3
* /mavros/fake_gps/geo_origin/alt: 408.0
* /mavros/fake_gps/geo_origin/lat: 47.3667
* /mavros/fake_gps/geo_origin/lon: 8.55
* /mavros/fake_gps/gps_id: 4
* /mavros/fake_gps/gps_rate: 5.0
* /mavros/fake_gps/horiz_accuracy: 0.5
* /mavros/fake_gps/mocap_transform: False
* /mavros/fake_gps/mocap_withcovariance: False
* /mavros/fake_gps/satellites_visible: 6
* /mavros/fake_gps/speed_accuracy: 0.0
* /mavros/fake_gps/tf/child_frame_id: fix
* /mavros/fake_gps/tf/frame_id: map
* /mavros/fake_gps/tf/listen: False
* /mavros/fake_gps/tf/rate_limit: 10.0
* /mavros/fake_gps/tf/send: False
* /mavros/fake_gps/use_hil_gps: True
* /mavros/fake_gps/use_mocap: True
* /mavros/fake_gps/use_vision: False
* /mavros/fake_gps/vert_accuracy: 0.5
* /mavros/fcu_protocol: v2.0
* /mavros/fcu_url: /dev/ttyAMA0
* /mavros/gcs_url:
* /mavros/global_position/child_frame_id: base_link
* /mavros/global_position/frame_id: map
* /mavros/global_position/gps_uere: 1.0
* /mavros/global_position/rot_covariance: 99999.0
* /mavros/global_position/tf/child_frame_id: base_link
* /mavros/global_position/tf/frame_id: map
* /mavros/global_position/tf/global_frame_id: earth
* /mavros/global_position/tf/send: False
* /mavros/global_position/use_relative_alt: True
* /mavros/image/frame_id: px4flow
* /mavros/imu/angular_velocity_stdev: 0.0003490659 // 0...
* /mavros/imu/frame_id: base_link
* /mavros/imu/linear_acceleration_stdev: 0.0003
* /mavros/imu/magnetic_stdev: 0.0
* /mavros/imu/orientation_stdev: 1.0
* /mavros/landing_target/camera/fov_x: 2.0071286398
* /mavros/landing_target/camera/fov_y: 2.0071286398
* /mavros/landing_target/image/height: 480
* /mavros/landing_target/image/width: 640
* /mavros/landing_target/land_target_type: VISION_FIDUCIAL
* /mavros/landing_target/listen_lt: False
* /mavros/landing_target/mav_frame: LOCAL_NED
* /mavros/landing_target/target_size/x: 0.3
* /mavros/landing_target/target_size/y: 0.3
* /mavros/landing_target/tf/child_frame_id: camera_center
* /mavros/landing_target/tf/frame_id: landing_target
* /mavros/landing_target/tf/listen: False
* /mavros/landing_target/tf/rate_limit: 10.0
* /mavros/landing_target/tf/send: True
* /mavros/local_position/frame_id: map
* /mavros/local_position/tf/child_frame_id: base_link
* /mavros/local_position/tf/frame_id: map
* /mavros/local_position/tf/send: False
* /mavros/local_position/tf/send_fcu: False
* /mavros/mission/pull_after_gcs: True
* /mavros/mission/use_mission_item_int: True
* /mavros/mocap/use_pose: True
* /mavros/mocap/use_tf: False
* /mavros/odometry/estimator_type: 3
* /mavros/odometry/frame_tf/desired_frame: ned
* /mavros/plugin_blacklist: ['actuator_contro...
* /mavros/plugin_whitelist: []
* /mavros/px4flow/frame_id: px4flow
* /mavros/px4flow/ranger_fov: 0.118682
* /mavros/px4flow/ranger_max_range: 5.0
* /mavros/px4flow/ranger_min_range: 0.3
* /mavros/safety_area/p1/x: 1.0
* /mavros/safety_area/p1/y: 1.0
* /mavros/safety_area/p1/z: 1.0
* /mavros/safety_area/p2/x: -1.0
* /mavros/safety_area/p2/y: -1.0
* /mavros/safety_area/p2/z: -1.0
* /mavros/setpoint_accel/send_force: False
* /mavros/setpoint_attitude/reverse_thrust: False
* /mavros/setpoint_attitude/tf/child_frame_id: target_attitude
* /mavros/setpoint_attitude/tf/frame_id: map
* /mavros/setpoint_attitude/tf/listen: False
* /mavros/setpoint_attitude/tf/rate_limit: 50.0
* /mavros/setpoint_attitude/use_quaternion: False
* /mavros/setpoint_position/mav_frame: LOCAL_NED
* /mavros/setpoint_position/tf/child_frame_id: target_position
* /mavros/setpoint_position/tf/frame_id: map
* /mavros/setpoint_position/tf/listen: False
* /mavros/setpoint_position/tf/rate_limit: 50.0
* /mavros/setpoint_raw/thrust_scaling: 1.0
* /mavros/setpoint_velocity/mav_frame: LOCAL_NED
* /mavros/startup_px4_usb_quirk: False
* /mavros/sys/disable_diag: False
* /mavros/sys/min_voltage: 10.0
* /mavros/target_component_id: 1
* /mavros/target_system_id: 1
* /mavros/tdr_radio/low_rssi: 40
* /mavros/time/time_ref_source: fcu
* /mavros/time/timesync_avg_alpha: 0.6
* /mavros/time/timesync_mode: MAVLINK
* /mavros/vibration/frame_id: base_link
* /mavros/vision_pose/tf/child_frame_id: vision_estimate
* /mavros/vision_pose/tf/frame_id: map
* /mavros/vision_pose/tf/listen: False
* /mavros/vision_pose/tf/rate_limit: 10.0
* /mavros/vision_speed/listen_twist: True
* /mavros/vision_speed/twist_cov: True
* /mavros/wheel_odometry/child_frame_id: base_link
* /mavros/wheel_odometry/count: 2
* /mavros/wheel_odometry/frame_id: map
* /mavros/wheel_odometry/send_raw: True
* /mavros/wheel_odometry/send_twist: False
* /mavros/wheel_odometry/tf/child_frame_id: base_link
* /mavros/wheel_odometry/tf/frame_id: map
* /mavros/wheel_odometry/tf/send: True
* /mavros/wheel_odometry/use_rpm: False
* /mavros/wheel_odometry/vel_error: 0.1
* /mavros/wheel_odometry/wheel0/radius: 0.05
* /mavros/wheel_odometry/wheel0/x: 0.0
* /mavros/wheel_odometry/wheel0/y: -0.15
* /mavros/wheel_odometry/wheel1/radius: 0.05
* /mavros/wheel_odometry/wheel1/x: 0.0
* /mavros/wheel_odometry/wheel1/y: 0.15
* /rosdistro: melodic
* /rosversion: 1.14.13
NODES
/
mavros (mavros/mavros_node)
auto-starting new master
process[master]: started with pid [24041]
ROS_MASTER_URI=http://localhost:11311
setting /run_id to cc6eafc6-bf23-11ec-9a31-dca63200a718
process[rosout-1]: started with pid [24052]
started core service [/rosout]
process[mavros-2]: started with pid [24060]
[ INFO] [1650292105.320250811]: FCU URL: /dev/ttyAMA0
[ INFO] [1650292105.333873660]: serial0: device: /dev/ttyAMA0 @ 57600 bps
[ INFO] [1650292105.334899222]: GCS bridge disabled
[ INFO] [1650292105.377218568]: Plugin 3dr_radio loaded
[ INFO] [1650292105.382715784]: Plugin 3dr_radio initialized
[ INFO] [1650292105.383287167]: Plugin actuator_control blacklisted
[ INFO] [1650292105.408167002]: Plugin adsb loaded
[ INFO] [1650292105.417653545]: Plugin adsb initialized
[ INFO] [1650292105.418070614]: Plugin altitude blacklisted
[ INFO] [1650292105.419167991]: Plugin cam_imu_sync loaded
[ INFO] [1650292105.421642704]: Plugin cam_imu_sync initialized
[ INFO] [1650292105.422749618]: Plugin camera loaded
[ INFO] [1650292105.425315664]: Plugin camera initialized
[ INFO] [1650292105.426622945]: Plugin command loaded
[ INFO] [1650292105.449871428]: Plugin command initialized
[ INFO] [1650292105.451007879]: Plugin companion_process_status loaded
[ INFO] [1650292105.457835395]: Plugin companion_process_status initialized
[ INFO] [1650292105.458192132]: Plugin debug_value blacklisted
[ INFO] [1650292105.459367971]: Plugin distance_sensor loaded
mavros_node: /usr/include/eigen3/Eigen/src/Core/DenseStorage.h:109: Eigen::internal::plain_array<T, Size, MatrixOrArrayOptions, 16>::plain_array() [with T = float; int Size = 4; int MatrixOrArrayOptions = 0]: Assertion `(internal::UIntPtr(eigen_unaligned_array_assert_workaround_gcc47(array)) & (15)) == 0 && "this assertion is explained here: " "http://eigen.tuxfamily.org/dox-devel/group__TopicUnalignedArrayAssert.html" " **** READ THIS WEB PAGE !!! ****"' failed.
================================================================================REQUIRED process [mavros-2] has died!
process has died [pid 24060, exit code -6, cmd /home/pi/ros_catkin_ws/devel/lib/mavros/mavros_node __name:=mavros __log:=/home/pi/.ros/log/cc6eafc6-bf23-11ec-9a31-dca63200a718/mavros-2.log].
log file: /home/pi/.ros/log/cc6eafc6-bf23-11ec-9a31-dca63200a718/mavros-2*.log
Initiating shutdown!
================================================================================
[mavros-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
Issue details
Segfault on startup causing mavros_node to die. Problem is only seen on Raspberry Pi 3, doesn't happen on laptop.
Problem is intermittent, happens approximately 70% of the time when I launch mavros.
I have built with
catkin build --cmake-args -DCMAKE_BUILD_TYPE=Debugand run with GDB, but the segfault backtrace only shows that the problem occurs inld-linux-armhf.so.3and doesnt show the trace into mavros source.MAVROS version and platform
Mavros: 0.21.1 Built from source ROS: Kinetic Ubuntu: 16.04
Autopilot type and version
[ x ] ArduPilot [ ] PX4
Version: 3.5
Node logs
Diagnostics
Check ID
No output when problem occurs