Closed davids5 closed 7 years ago
@davids5 It does use usleep under the hood. That's probably just a bad idea for CI. We'll need to mock time at some point for the unit tests.
Alright, I'll take a look at that hysteresis test when I have some time. @LorenzMeier or @julianoes are you seeing intermittent hysteresis test failures anywhere else?
Overall the only way this gets better is by not tolerating intermittent failures and weird CI behaviour. Please be super annoying and add a comment here every time you see a failure like this. One line summary and the build link.
With the posix daemon (https://github.com/PX4/Firmware/pull/5162) stuff it should be possible to split up the tests, which means we can start gathering test reports to isolate particular failures over time. Circleci already has support for this if we convert the ctest output to junit style.
@dager
This build would not finish
BFormat checks passed
Error deleting container: Error response from daemon: Driver btrfs failed to remove root filesystem 03f2fe2d46a055406cc9ae319dadf840b3999bdfe2db721ed6b962cd9b8a9a71: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 03f2fe2d46a055406cc9ae319dadf840b3999bdfe2db721ed6b962cd9b8a9a71: operation not permitted
Had to cancel it to get Success!
@dagar @LorenzMeier
I have an idea that I think will speed up the CI build a lot!
I makes the most sense for CI and not the regular build be cause of the dependency checks for parameters.
If we split the parameter generation into 2 pieces. The first does the complete walk of the tree and builds an xml file of all the parameters and saves it to a parameter.cache.xml.
The second pass uses the parameter.cache.xml. file if it exists and skips that part of the process and on to do the culling and parameter generation for each build based on the modules in the cmake.
That circleci error has something to do with their docker setup. I'll contact them.
I suggest we time how long each phase is actually taking before we try any build optimization. To me it looks like param generation is only a few seconds per build.
@dagar
Here is an Example of the the phantom errors related to the built ins:
/lib/ccache/arm-none-eabi-g++ -g -fno-exceptions -fno-rtti -std=gnu++0x -fno-threadsafe-statics -DCONFIG_WCHAR_BUILTIN -D__CUSTOM_FILE_IO__ -fcheck-new -Wall -Werror -Wextra -Wno-sign-compare -Wshadow -Wfloat-equal -Wpointer-arith -Wmissing-declarations -Wno-unused-parameter -Werror=format-security -Werror=array-bounds -Wfatal-errors -Werror=unused-variable -Werror=reorder -Werror=uninitialized -Werror=init-self -Werror=unused-but-set-variable -Wformat=1 -Wdouble-promotion -Werror=double-promotion -Wno-missing-field-initializers -fvisibility=hidden -include visibility.h -mcpu=cortex-m4 -mthumb -march=armv7e-m -mfpu=fpv4-sp-d16 -mfloat-abi=hard -nodefaultlibs -nostdlib -Os -fno-strict-aliasing -fomit-frame-pointer -funsafe-math-optimizations -ffunction-sections -fdata-sections -fno-strength-reduce -fno-builtin-printf -O2 -g -DNDEBUG -Wl,--warn-common -Wl,--gc-sections -T/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/build/ld.script -Wl,-Map=/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/main.map src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj -o src/firmware/nuttx/firmware_nuttx -L/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs -L/home/runner/Firmware/build_mindpx-v2_default/external/Install/lib -Wl,--warn-common -Wl,--gc-sections -Wl,--start-group src/drivers/device/libdrivers__device.a src/drivers/stm32/libdrivers__stm32.a src/drivers/stm32/adc/libdrivers__stm32__adc.a src/drivers/stm32/tone_alarm/libdrivers__stm32__tone_alarm.a src/drivers/led/libdrivers__led.a src/drivers/px4fmu/libdrivers__px4fmu.a src/drivers/boards/mindpx-v2/libdrivers__boards__mindpx-v2.a src/drivers/rgbled_pwm/libdrivers__rgbled_pwm.a src/drivers/mpu6500/libdrivers__mpu6500.a src/drivers/lsm303d/libdrivers__lsm303d.a src/drivers/l3gd20/libdrivers__l3gd20.a src/drivers/hmc5883/libdrivers__hmc5883.a src/drivers/ms5611/libdrivers__ms5611.a src/drivers/srf02_i2c/libdrivers__srf02_i2c.a src/drivers/hc_sr04/libdrivers__hc_sr04.a src/drivers/gps/libdrivers__gps.a src/drivers/pwm_out_sim/libdrivers__pwm_out_sim.a src/drivers/blinkm/libdrivers__blinkm.a src/drivers/airspeed/libdrivers__airspeed.a src/drivers/ets_airspeed/libdrivers__ets_airspeed.a src/drivers/meas_airspeed/libdrivers__meas_airspeed.a src/drivers/frsky_telemetry/libdrivers__frsky_telemetry.a src/modules/sensors/libmodules__sensors.a src/drivers/px4flow/libdrivers__px4flow.a src/drivers/vmount/libdrivers__vmount.a src/drivers/pwm_input/libdrivers__pwm_input.a src/drivers/camera_trigger/libdrivers__camera_trigger.a src/drivers/bst/libdrivers__bst.a src/drivers/snapdragon_rc_pwm/libdrivers__snapdragon_rc_pwm.a src/systemcmds/bl_update/libsystemcmds__bl_update.a src/systemcmds/mixer/libsystemcmds__mixer.a src/systemcmds/param/libsystemcmds__param.a src/systemcmds/perf/libsystemcmds__perf.a src/systemcmds/pwm/libsystemcmds__pwm.a src/systemcmds/esc_calib/libsystemcmds__esc_calib.a src/systemcmds/hardfault_log/libsystemcmds__hardfault_log.a src/systemcmds/reboot/libsystemcmds__reboot.a src/systemcmds/topic_listener/libsystemcmds__topic_listener.a src/systemcmds/top/libsystemcmds__top.a src/systemcmds/config/libsystemcmds__config.a src/systemcmds/nshterm/libsystemcmds__nshterm.a src/systemcmds/mtd/libsystemcmds__mtd.a src/systemcmds/dumpfile/libsystemcmds__dumpfile.a src/systemcmds/ver/libsystemcmds__ver.a src/systemcmds/sd_bench/libsystemcmds__sd_bench.a src/systemcmds/motor_ramp/libsystemcmds__motor_ramp.a src/drivers/sf0x/sf0x_tests/libdrivers__sf0x__sf0x_tests.a src/drivers/test_ppm/libdrivers__test_ppm.a src/modules/commander/commander_tests/libmodules__commander__commander_tests.a src/modules/mc_pos_control/mc_pos_control_tests/libmodules__mc_pos_control__mc_pos_control_tests.a src/modules/controllib_test/libmodules__controllib_test.a src/modules/mavlink/mavlink_tests/libmodules__mavlink__mavlink_tests.a src/modules/unit_test/libmodules__unit_test.a src/modules/uORB/uORB_tests/libmodules__uORB__uORB_tests.a src/systemcmds/tests/libsystemcmds__tests.a src/modules/commander/libmodules__commander.a src/modules/load_mon/libmodules__load_mon.a src/modules/navigator/libmodules__navigator.a src/modules/mavlink/libmodules__mavlink.a src/modules/gpio_led/libmodules__gpio_led.a src/modules/uavcan/libmodules__uavcan.a src/modules/land_detector/libmodules__land_detector.a src/modules/attitude_estimator_q/libmodules__attitude_estimator_q.a src/modules/position_estimator_inav/libmodules__position_estimator_inav.a src/modules/local_position_estimator/libmodules__local_position_estimator.a src/modules/ekf2/libmodules__ekf2.a src/modules/fw_pos_control_l1/libmodules__fw_pos_control_l1.a src/modules/fw_att_control/libmodules__fw_att_control.a src/modules/mc_att_control/libmodules__mc_att_control.a src/modules/mc_pos_control/libmodules__mc_pos_control.a src/modules/vtol_att_control/libmodules__vtol_att_control.a src/modules/sdlog2/libmodules__sdlog2.a src/modules/logger/libmodules__logger.a src/modules/param/libmodules__param.a src/modules/systemlib/libmodules__systemlib.a src/modules/systemlib/mixer/libmodules__systemlib__mixer.a src/modules/uORB/libmodules__uORB.a src/modules/dataman/libmodules__dataman.a src/lib/controllib/liblib__controllib.a src/lib/mathlib/liblib__mathlib.a src/lib/mathlib/math/filter/liblib__mathlib__math__filter.a src/lib/ecl/liblib__ecl.a src/lib/external_lgpl/liblib__external_lgpl.a src/lib/geo/liblib__geo.a src/lib/geo_lookup/liblib__geo_lookup.a src/lib/conversion/liblib__conversion.a src/lib/launchdetection/liblib__launchdetection.a src/lib/terrain_estimation/liblib__terrain_estimation.a src/lib/runway_takeoff/liblib__runway_takeoff.a src/lib/tailsitter_recovery/liblib__tailsitter_recovery.a src/lib/DriverFramework/framework/liblib__DriverFramework__framework.a src/platforms/nuttx/libplatforms__nuttx.a src/platforms/common/libplatforms__common.a src/platforms/nuttx/px4_layer/libplatforms__nuttx__px4_layer.a src/modules/uavcan/libuavcan/libuavcan/libuavcan.a src/modules/uavcan/libuavcan/libuavcan_drivers/stm32/driver/libuavcan_stm32_driver.a -lapps -lnuttx -lm -lgcc src/firmware/nuttx/libromfs.a -Wl,--end-group libmsg_gen.a src/lib/DriverFramework/framework/src/libdf_driver_framework.a && :
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs/libapps.a(nsh_builtin.o): In function `nsh_builtin':
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/apps/nshlib/nsh_builtin.c:105: undefined reference to `exec_builtin'
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs/libnuttx.a(binfmt-libbuiltin_getname.o): In function `builtin_getname':
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx/binfmt/libbuiltin/libbuiltin_getname.c:88: undefined reference to `builtin_for_index'
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs/libnuttx.a(binfmt-builtin.o): In function `builtin_loadbinary':
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx/binfmt/builtin.c:141: undefined reference to `builtin_for_index'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Makefile:166: recipe for target 'mindpx-v2_default' failed
make[1]: *** [mindpx-v2_default] Error 1
Makefile:278: recipe for target 'check_mindpx-v2_default' failed
make: *** [check_mindpx-v2_default] Error 2
and https://travis-ci.org/PX4/Firmware/jobs/170321947
Building px4nucleoF767ZI-v1_default
GIT_SUBMODULES_ARE_EVIL is defined - Skipping All submodule checking!
-- nuttx-px4nucleoF767ZI-v1-default
-- CMAKE_INSTALL_PREFIX:
-- cmake module path: /home/travis/build/PX4/Firmware/cmake
-- Nuttx build for px4nucleoF767ZI-v1 on m7 hardware, using nsh with ROMFS on px4fmu_common
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib/ccache/arm-none-eabi-gcc
-- catkin DISABLED
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7")
-- NuttX patch: nuttx-patches/Fixed-Shadow-wanings.patch
-- NuttX patch: nuttx-patches/nsh_parse.patch
-- NuttX patch: nuttx-patches/c++11.patch
-- NuttX patch: nuttx-patches/wip_inflight_to_upstream.patch
-- NuttX patch: nuttx-patches/silence-jobserver-warnings.patch
-- NuttX patch: nuttx-patches/static_asset-fix.patch
-- NuttX patch: nuttx-patches/ctype_cctype_.patch
-- NuttX patch: nuttx-patches/fixStdint.patch
-- NuttX patch: nuttx-patches/math.h.patch
-- Using C++03
-- Release build type: RelWithDebInfo
-- Adding UAVCAN STM32 platform driver
-- Adding ROMFS on px4nucleoF767ZI-v1
-- GIT_TAG = v1.5.0-1959-gf5b9cdb
-- Configuring done
-- Generating done
-- Build files have been written to: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
PX4 CONFIG: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
ninja: Entering directory `/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default'
[542/543] Linking CXX executable src/firmware/nuttx/firmware_nuttx
FAILED: : && /usr/lib/ccache/arm-none-eabi-g++ -g -fno-exceptions -fno-rtti -std=gnu++0x -fno-threadsafe-statics -DCONFIG_WCHAR_BUILTIN -D__CUSTOM_FILE_IO__ -fcheck-new -Wall -Werror -Wextra -Wno-sign-compare -Wshadow -Wfloat-equal -Wpointer-arith -Wmissing-declarations -Wno-unused-parameter -Werror=format-security -Werror=array-bounds -Wfatal-errors -Werror=unused-variable -Werror=reorder -Werror=uninitialized -Werror=init-self -Werror=unused-but-set-variable -Wformat=1 -Wdouble-promotion -Werror=double-promotion -Wno-missing-field-initializers -fvisibility=hidden -include visibility.h -mcpu=cortex-m7 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=hard -nodefaultlibs -nostdlib -Os -fno-strict-aliasing -fomit-frame-pointer -funsafe-math-optimizations -ffunction-sections -fdata-sections -fno-strength-reduce -fno-builtin-printf -O2 -g -DNDEBUG -Wl,--warn-common -Wl,--gc-sections -T/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/build/ld.script -Wl,-Map=/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/main.map src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj -o src/firmware/nuttx/firmware_nuttx -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/libs -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/external/Install/lib -Wl,--warn-common -Wl,--gc-sections -Wl,--start-group src/drivers/device/libdrivers__device.a src/drivers/stm32/libdrivers__stm32.a src/drivers/stm32/adc/libdrivers__stm32__adc.a src/drivers/stm32/tone_alarm/libdrivers__stm32__tone_alarm.a src/drivers/led/libdrivers__led.a src/drivers/px4fmu/libdrivers__px4fmu.a src/drivers/boards/px4nucleoF767ZI-v1/libdrivers__boards__px4nucleoF767ZI-v1.a src/drivers/rgbled/libdrivers__rgbled.a src/drivers/mpu6000/libdrivers__mpu6000.a src/drivers/mpu9250/libdrivers__mpu9250.a src/drivers/hmc5883/libdrivers__hmc5883.a src/drivers/ms5611/libdrivers__ms5611.a src/drivers/mb12xx/libdrivers__mb12xx.a src/drivers/srf02/libdrivers__srf02.a src/drivers/sf0x/libdrivers__sf0x.a src/drivers/ll40ls/libdrivers__ll40ls.a src/drivers/trone/libdrivers__trone.a src/drivers/gps/libdrivers__gps.a src/drivers/pwm_out_sim/libdrivers__pwm_out_sim.a src/drivers/hott/libdrivers__hott.a src/drivers/hott/hott_telemetry/libdrivers__hott__hott_telemetry.a src/drivers/hott/hott_sensors/libdrivers__hott__hott_sensors.a src/drivers/blinkm/libdrivers__blinkm.a src/drivers/airspeed/libdrivers__airspeed.a src/drivers/ets_airspeed/libdrivers__ets_airspeed.a src/drivers/meas_airspeed/libdrivers__meas_airspeed.a src/drivers/frsky_telemetry/libdrivers__frsky_telemetry.a src/modules/sensors/libmodules__sensors.a src/drivers/mkblctrl/libdrivers__mkblctrl.a src/drivers/px4flow/libdrivers__px4flow.a src/drivers/oreoled/libdrivers__oreoled.a src/drivers/pwm_input/libdrivers__pwm_input.a src/drivers/camera_trigger/libdrivers__camera_trigger.a src/drivers/bst/libdrivers__bst.a src/drivers/snapdragon_rc_pwm/libdrivers__snapdragon_rc_pwm.a src/drivers/lis3mdl/libdrivers__lis3mdl.a src/drivers/bmp280/libdrivers__bmp280.a src/drivers/bma180/libdrivers__bma180.a src/drivers/bmi160/libdrivers__bmi160.a src/drivers/tap_esc/libdrivers__tap_esc.a src/systemcmds/bl_update/libsystemcmds__bl_update.a src/systemcmds/config/libsystemcmds__config.a src/systemcmds/dumpfile/libsystemcmds__dumpfile.a src/systemcmds/esc_calib/libsystemcmds__esc_calib.a src/systemcmds/hardfault_log/libsystemcmds__hardfault_log.a src/systemcmds/mixer/libsystemcmds__mixer.a src/systemcmds/motor_ramp/libsystemcmds__motor_ramp.a src/systemcmds/mtd/libsystemcmds__mtd.a src/systemcmds/nshterm/libsystemcmds__nshterm.a src/systemcmds/param/libsystemcmds__param.a src/systemcmds/perf/libsystemcmds__perf.a src/systemcmds/pwm/libsystemcmds__pwm.a src/systemcmds/reboot/libsystemcmds__reboot.a src/systemcmds/sd_bench/libsystemcmds__sd_bench.a src/systemcmds/top/libsystemcmds__top.a src/systemcmds/topic_listener/libsystemcmds__topic_listener.a src/systemcmds/ver/libsystemcmds__ver.a src/modules/commander/libmodules__commander.a src/modules/load_mon/libmodules__load_mon.a src/modules/navigator/libmodules__navigator.a src/modules/mavlink/libmodules__mavlink.a src/modules/gpio_led/libmodules__gpio_led.a src/modules/uavcan/libmodules__uavcan.a src/modules/land_detector/libmodules__land_detector.a src/modules/attitude_estimator_q/libmodules__attitude_estimator_q.a src/modules/position_estimator_inav/libmodules__position_estimator_inav.a src/modules/local_position_estimator/libmodules__local_position_estimator.a src/modules/ekf2/libmodules__ekf2.a src/modules/fw_pos_control_l1/libmodules__fw_pos_control_l1.a src/modules/fw_att_control/libmodules__fw_att_control.a src/modules/mc_att_control/libmodules__mc_att_control.a src/modules/mc_pos_control/libmodules__mc_pos_control.a src/modules/vtol_att_control/libmodules__vtol_att_control.a src/modules/logger/libmodules__logger.a src/modules/sdlog2/libmodules__sdlog2.a src/modules/param/libmodules__param.a src/modules/systemlib/libmodules__systemlib.a src/modules/systemlib/mixer/libmodules__systemlib__mixer.a src/modules/uORB/libmodules__uORB.a src/modules/dataman/libmodules__dataman.a src/lib/controllib/liblib__controllib.a src/lib/mathlib/liblib__mathlib.a src/lib/mathlib/math/filter/liblib__mathlib__math__filter.a src/lib/rc/liblib__rc.a src/lib/ecl/liblib__ecl.a src/lib/external_lgpl/liblib__external_lgpl.a src/lib/geo/liblib__geo.a src/lib/geo_lookup/liblib__geo_lookup.a src/lib/conversion/liblib__conversion.a src/lib/launchdetection/liblib__launchdetection.a src/lib/terrain_estimation/liblib__terrain_estimation.a src/lib/runway_takeoff/liblib__runway_takeoff.a src/lib/tailsitter_recovery/liblib__tailsitter_recovery.a src/lib/DriverFramework/framework/liblib__DriverFramework__framework.a src/platforms/nuttx/libplatforms__nuttx.a src/platforms/common/libplatforms__common.a src/platforms/nuttx/px4_layer/libplatforms__nuttx__px4_layer.a src/modules/bottle_drop/libmodules__bottle_drop.a src/examples/rover_steering_control/libexamples__rover_steering_control.a src/examples/px4_simple_app/libexamples__px4_simple_app.a src/modules/uavcan/libuavcan/libuavcan/libuavcan.a src/modules/uavcan/libuavcan/libuavcan_drivers/stm32/driver/libuavcan_stm32_driver.a -lapps -lnuttx -lm -lgcc src/firmware/nuttx/libromfs.a -Wl,--end-group libmsg_gen.a src/lib/DriverFramework/framework/src/libdf_driver_framework.a && :
src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj:(.rodata.g_builtins+0x4ac): undefined reference to `serdis_main'
src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj:(.rodata.g_builtins+0x4bc): undefined reference to `sercon_main'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Makefile:166: recipe for target 'px4nucleoF767ZI-v1_default' failed
make[1]: *** [px4nucleoF767ZI-v1_default] Error 1
Makefile:278: recipe for target 'check_px4nucleoF767ZI-v1_default' failed
make: *** [check_px4nucleoF767ZI-v1_default] Error 2
The command "if [ "${TRAVIS_OS_NAME}" = "linux" ]; then docker run --rm -v `pwd`:`pwd`:rw -v $HOME/.ccache:$HOME/.ccache:rw -e CCACHE_DIR=$HOME/.ccache -e CI=true -e GIT_SUBMODULES_ARE_EVIL=1 -w=`pwd` --user=$UID -it ${DOCKER_REPO} /bin/bash -c "make check VECTORCONTROL=1"; elif [ "${TRAVIS_OS_NAME}" = "osx" ]; then make tests; fi" exited with 2.
Are you mainly seeing these on travis-ci, semaphore or locally? I feel like if it was happening in master someone would have mentioned it by now. I should take a look at your nuttx_v3 cmake changes.
@dagar
Only on CI.
Please do!
@dagar
This was the "Need more boxes" I had mentioned
I'm not following here but if the problem is only the usleeps in the hysteresis check, let's disable it until we have mocked time.
@julianoes this has become a bit of a catch all for CI failures that @davids5's been hitting.
@dagar more of the same
Building px4nucleoF767ZI-v1_default
GIT_SUBMODULES_ARE_EVIL is defined - Skipping All submodule checking!
-- nuttx-px4nucleoF767ZI-v1-default
-- CMAKE_INSTALL_PREFIX:
-- cmake module path: /home/travis/build/PX4/Firmware/cmake
-- Nuttx build for px4nucleoF767ZI-v1 on m7 hardware, using nsh with ROMFS on px4fmu_common
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib/ccache/arm-none-eabi-gcc
-- catkin DISABLED
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7")
-- NuttX patch: nuttx-patches/Fixed-Shadow-wanings.patch
-- NuttX patch: nuttx-patches/nsh_parse.patch
-- NuttX patch: nuttx-patches/c++11.patch
-- NuttX patch: nuttx-patches/wip_inflight_to_upstream.patch
-- NuttX patch: nuttx-patches/silence-jobserver-warnings.patch
-- NuttX patch: nuttx-patches/static_asset-fix.patch
-- NuttX patch: nuttx-patches/wip_nxphlite_nuttx_cfg.patch
-- NuttX patch: nuttx-patches/ctype_cctype_.patch
-- NuttX patch: nuttx-patches/fixStdint.patch
-- NuttX patch: nuttx-patches/math.h.patch
-- Using C++03
-- Release build type: RelWithDebInfo
-- Adding UAVCAN STM32 platform driver
-- Adding ROMFS on px4nucleoF767ZI-v1
-- GIT_TAG = v1.5.0-1979-g866ec5f
-- Configuring done
-- Generating done
-- Build files have been written to: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
PX4 CONFIG: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
ninja: Entering directory `/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default'
[543/544] Linking CXX executable src/firmware/nuttx/firmware_nuttx
FAILED: : && /usr/lib/ccache/arm-none-eabi-g++ -g -fno-exceptions -fno-rtti -std=gnu++0x -fno-threadsafe-statics -DCONFIG_WCHAR_BUILTIN -D__CUSTOM_FILE_IO__ -fcheck-new -Wall -Werror -Wextra -Wno-sign-compare -Wshadow -Wfloat-equal -Wpointer-arith -Wmissing-declarations -Wno-unused-parameter -Werror=format-security -Werror=array-bounds -Wfatal-errors -Werror=unused-variable -Werror=reorder -Werror=uninitialized -Werror=init-self -Werror=unused-but-set-variable -Wformat=1 -Wdouble-promotion -Werror=double-promotion -Wno-missing-field-initializers -fvisibility=hidden -include visibility.h -mcpu=cortex-m7 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=hard -nodefaultlibs -nostdlib -Os -fno-strict-aliasing -fomit-frame-pointer -funsafe-math-optimizations -ffunction-sections -fdata-sections -fno-strength-reduce -fno-builtin-printf -O2 -g -DNDEBUG -Wl,--warn-common -Wl,--gc-sections -T/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/build/ld.script -Wl,-Map=/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/main.map src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj -o src/firmware/nuttx/firmware_nuttx -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/libs -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/external/Install/lib -Wl,--warn-common -Wl,--gc-sections -Wl,--start-group src/drivers/device/libdrivers__device.a src/drivers/stm32/libdrivers__stm32.a src/drivers/stm32/adc/libdrivers__stm32__adc.a src/drivers/stm32/tone_alarm/libdrivers__stm32__tone_alarm.a src/drivers/led/libdrivers__led.a src/drivers/px4fmu/libdrivers__px4fmu.a src/drivers/boards/px4nucleoF767ZI-v1/libdrivers__boards__px4nucleoF767ZI-v1.a src/drivers/rgbled/libdrivers__rgbled.a src/drivers/mpu6000/libdrivers__mpu6000.a src/drivers/mpu9250/libdrivers__mpu9250.a src/drivers/hmc5883/libdrivers__hmc5883.a src/drivers/ms5611/libdrivers__ms5611.a src/drivers/mb12xx/libdrivers__mb12xx.a src/drivers/srf02/libdrivers__srf02.a src/drivers/sf0x/libdrivers__sf0x.a src/drivers/ll40ls/libdrivers__ll40ls.a src/drivers/trone/libdrivers__trone.a src/drivers/gps/libdrivers__gps.a src/drivers/pwm_out_sim/libdrivers__pwm_out_sim.a src/drivers/hott/libdrivers__hott.a src/drivers/hott/hott_telemetry/libdrivers__hott__hott_telemetry.a src/drivers/hott/hott_sensors/libdrivers__hott__hott_sensors.a src/drivers/blinkm/libdrivers__blinkm.a src/drivers/airspeed/libdrivers__airspeed.a src/drivers/ets_airspeed/libdrivers__ets_airspeed.a src/drivers/meas_airspeed/libdrivers__meas_airspeed.a src/drivers/frsky_telemetry/libdrivers__frsky_telemetry.a src/modules/sensors/libmodules__sensors.a src/drivers/mkblctrl/libdrivers__mkblctrl.a src/drivers/px4flow/libdrivers__px4flow.a src/drivers/oreoled/libdrivers__oreoled.a src/drivers/pwm_input/libdrivers__pwm_input.a src/drivers/camera_trigger/libdrivers__camera_trigger.a src/drivers/bst/libdrivers__bst.a src/drivers/snapdragon_rc_pwm/libdrivers__snapdragon_rc_pwm.a src/drivers/lis3mdl/libdrivers__lis3mdl.a src/drivers/bmp280/libdrivers__bmp280.a src/drivers/bma180/libdrivers__bma180.a src/drivers/bmi160/libdrivers__bmi160.a src/drivers/tap_esc/libdrivers__tap_esc.a src/systemcmds/bl_update/libsystemcmds__bl_update.a src/systemcmds/config/libsystemcmds__config.a src/systemcmds/dumpfile/libsystemcmds__dumpfile.a src/systemcmds/esc_calib/libsystemcmds__esc_calib.a src/systemcmds/hardfault_log/libsystemcmds__hardfault_log.a src/systemcmds/mixer/libsystemcmds__mixer.a src/systemcmds/motor_ramp/libsystemcmds__motor_ramp.a src/systemcmds/mtd/libsystemcmds__mtd.a src/systemcmds/nshterm/libsystemcmds__nshterm.a src/systemcmds/param/libsystemcmds__param.a src/systemcmds/perf/libsystemcmds__perf.a src/systemcmds/pwm/libsystemcmds__pwm.a src/systemcmds/reboot/libsystemcmds__reboot.a src/systemcmds/sd_bench/libsystemcmds__sd_bench.a src/systemcmds/top/libsystemcmds__top.a src/systemcmds/topic_listener/libsystemcmds__topic_listener.a src/systemcmds/ver/libsystemcmds__ver.a src/modules/commander/libmodules__commander.a src/modules/load_mon/libmodules__load_mon.a src/modules/navigator/libmodules__navigator.a src/modules/mavlink/libmodules__mavlink.a src/modules/gpio_led/libmodules__gpio_led.a src/modules/uavcan/libmodules__uavcan.a src/modules/land_detector/libmodules__land_detector.a src/modules/attitude_estimator_q/libmodules__attitude_estimator_q.a src/modules/position_estimator_inav/libmodules__position_estimator_inav.a src/modules/local_position_estimator/libmodules__local_position_estimator.a src/modules/ekf2/libmodules__ekf2.a src/modules/fw_pos_control_l1/libmodules__fw_pos_control_l1.a src/modules/fw_att_control/libmodules__fw_att_control.a src/modules/mc_att_control/libmodules__mc_att_control.a src/modules/mc_pos_control/libmodules__mc_pos_control.a src/modules/vtol_att_control/libmodules__vtol_att_control.a src/modules/logger/libmodules__logger.a src/modules/sdlog2/libmodules__sdlog2.a src/modules/param/libmodules__param.a src/modules/systemlib/libmodules__systemlib.a src/modules/systemlib/mixer/libmodules__systemlib__mixer.a src/modules/uORB/libmodules__uORB.a src/modules/dataman/libmodules__dataman.a src/lib/controllib/liblib__controllib.a src/lib/mathlib/liblib__mathlib.a src/lib/mathlib/math/filter/liblib__mathlib__math__filter.a src/lib/rc/liblib__rc.a src/lib/ecl/liblib__ecl.a src/lib/external_lgpl/liblib__external_lgpl.a src/lib/geo/liblib__geo.a src/lib/geo_lookup/liblib__geo_lookup.a src/lib/conversion/liblib__conversion.a src/lib/launchdetection/liblib__launchdetection.a src/lib/terrain_estimation/liblib__terrain_estimation.a src/lib/runway_takeoff/liblib__runway_takeoff.a src/lib/tailsitter_recovery/liblib__tailsitter_recovery.a src/lib/DriverFramework/framework/liblib__DriverFramework__framework.a src/platforms/nuttx/libplatforms__nuttx.a src/platforms/common/libplatforms__common.a src/platforms/nuttx/px4_layer/libplatforms__nuttx__px4_layer.a src/modules/bottle_drop/libmodules__bottle_drop.a src/examples/rover_steering_control/libexamples__rover_steering_control.a src/examples/px4_simple_app/libexamples__px4_simple_app.a src/modules/uavcan/libuavcan/libuavcan/libuavcan.a src/modules/uavcan/libuavcan/libuavcan_drivers/stm32/driver/libuavcan_stm32_driver.a -lapps -lnuttx -lm -lgcc src/firmware/nuttx/libromfs.a -Wl,--end-group libmsg_gen.a src/lib/DriverFramework/framework/src/libdf_driver_framework.a && :
/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/libs/libapps.a(nsh_session.o): In function `nsh_session':
/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/apps/nshlib/nsh_session.c:148: undefined reference to `readline'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Makefile:166: recipe for target 'px4nucleoF767ZI-v1_default' failed
make[1]: *** [px4nucleoF767ZI-v1_default] Error 1
Makefile:279: recipe for target 'check_px4nucleoF767ZI-v1_default' failed
make: *** [check_px4nucleoF767ZI-v1_default] Error 2
It looks like our environment + builds don't leave enough room to compress the docker image. I've removed the docker snapshot from semaphoreci for now. It didn't save that much time.
@dagar more [hysteresis] FAIL
Was that a different build? I don't see that failure.
I restarted it
Ah okay, double fail.
@LorenzMeier said we are going to switch over to nuttx_v3 in the next 3 weeks or so. So we have better get the demons out.
Got it. I'll have a little bit of time to work through these issues later tonight.
@dagar thank you!
@dagar
https://travis-ci.org/PX4/Firmware/jobs/178478801
I knew I should have never dated Travis's sister :)
That's a somewhat regular failure where travis-ci can't reach github via http. Might be more reliable over ssh, but then it's slightly harder for people to access the repo.
@dagar We are going to need
arm-none-eabi-gcc (GNU Tools for ARM Embedded Processors) 5.4.1 20160609 (release) [ARM/embedded-5-branch revision 237715]
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
on semaphoreci.com
https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/16
Is there a way to install both 9.3 and 5.4.1 compilers and then have the make file or a script select it?
In this particular case for semaphoreci we can move it up to 5.4.1 by pointing it at a different docker image. GCC 4.9 stable builds for QGC are handled on travis-ci.
We could have larger images with both installed, but I'm not sure if there's a good way for the build system to select a version, and in fact it should probably always respect what the user/developer sets for CC and CXX.
If we have minimum version requirements can you enforce them in cmake? Perhaps here - https://github.com/PX4/Firmware/blob/master/cmake/nuttx/px4_impl_nuttx.cmake
@dagar please rev semaphoreci to 5.41 then.
Also Please do the OR for nuttx_v3 that does the enforcement. Should we have an override from the env to help with A/B testing?
@dagar
How can we fix this.
https://semaphoreci.com/px4/firmware/branches/nuttx_v3_fmuv5/builds/4
rsync: recv_generator: mkdir "/home/runner/Firmware/build_px4esc-v1_bootloader/px4esc-v1/NuttX/nuttx/wireless" failed: No space left on device (28)
I removed docker image caching from semaphoreci and wiped the existing cache. It's not clear how much space they give us per build. I restarted your build.
@dagar - still no joy!
https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/22
I only partially disabled ccache before. Trying again with it completely disabled and wiped. Build restarted - https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/22
@dagar - this is starting to remind me of a movie with zombies that can not be killed.
Fatal error: can't create at45db.o: No space left on device
I guess we're just running out of space with all the individual builds? Each has a copy of nuttx, build artifacts, etc. Can you add df -h
between each build or something?
@dagar
I nodded you expertise - I think that last parallelism change we made for CI broke a non nija build
3%] Copying NuttX for px4fmu-v2 with nsh
[ 3%] Copying NuttX for px4fmu-v2 with nsh
[ 3%] Copying NuttX for px4fmu-v2 with nsh
[ 3%] Built target xml_gen
Scanning dependencies of target nuttx_patch_ctype_cctype_.patch-px4fmu-v2
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" -> "NxWidgets/ChangeLog.txt": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" -> "apps/system/zmodem/zm_state.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" -> "misc/tools/kconfig-frontends/scripts/.autostuff/m4/libtool.m4": No such file or directory (2)
[ 3%] Copying NuttX for px4fmu-v2 with nsh
file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_ohciram.h.sE3Jqr"
file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_pwm.c.WZr2hd"
IO error encountered -- skipping file deletion
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" -> "nuttx/arch/arm/src/lpc17xx/lpc17_serial.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" -> "nuttx/arch/z16/src/z16f/chip.h": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" -> "nuttx/binfmt/binfmt.h": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" -> "nuttx/audio/audio.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" -> "nuttx/configs/lm4f120-launchpad/src/lm4f_autoleds.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" -> "nuttx/conf
How to break it
make clean && make NO_NINJA_BUILD=1 px4fmu-v2_default
How to fix it
make clean && make px4fmu-v2_default
Strange, I wonder if there's an assumption made about a relative path that's actually only valid for the generated ninja, but not make. A bit of a wild guess... I'll definitely find a way to fix this, but likely won't be able to until next week (currently travelling).
On Fri, Dec 16, 2016 at 2:36 PM, David Sidrane notifications@github.com wrote:
@dagar https://github.com/dagar
I nodded you expertise - I think that last parallelism change we made for CI broke a non nija build
3%] Copying NuttX for px4fmu-v2 with nsh [ 3%] Copying NuttX for px4fmu-v2 with nsh [ 3%] Copying NuttX for px4fmu-v2 with nsh [ 3%] Built target xml_gen Scanning dependencies of target nuttx_patch_ctypecctype.patch-px4fmu-v2 rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" -> "NxWidgets/ChangeLog.txt": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" -> "apps/system/zmodem/zm_state.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" -> "misc/tools/kconfig-frontends/scripts/.autostuff/m4/libtool.m4": No such file or directory (2) [ 3%] Copying NuttX for px4fmu-v2 with nsh file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_ohciram.h.sE3Jqr" file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_pwm.c.WZr2hd" IO error encountered -- skipping file deletion rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" -> "nuttx/arch/arm/src/lpc17xx/lpc17_serial.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" -> "nuttx/arch/z16/src/z16f/chip.h": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" -> "nuttx/binfmt/binfmt.h": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" -> "nuttx/audio/audio.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" -> "nuttx/configs/lm4f120-launchpad/src/lm4f_autoleds.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" -> "nuttx/conf
How to break it
make clean && make NO_NINJA_BUILD=1 px4fmu-v2_default
How to fix it
make clean && make px4fmu-v2_default
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/PX4/Firmware/issues/5717#issuecomment-267677937, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFK6KjRNxz6en4j7_rZ3_NzHDdwOyV3ks5rIuhDgaJpZM4KdWjv .
The cmake generated makefile is actually running the copy multiple times.
@davids5 should be resolved on master. Can you verify?
@dagar Capturing this here.
I have had many issues with odd CI failures - that get resolved by one of the two following remedies: A) Either restarting the CI or B) deleting the cache and restarting the CI.
Below is a summary:
I have seen this a bunch of times - timing?
https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/1
Remedy A) rixes it.
The next class I keep seeing is unresolved externals in travis. Usually related to a lot of builtin command symbols:
sercon serdis
In this case only remedy B) works.
Unfortunately I do not have a capture - but I will post one the next time it happens