search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.58k stars 13.56k forks source link

PX4 Continuous Integration False failures #5717

Closed davids5 closed 7 years ago

davids5 commented 8 years ago

@dagar Capturing this here.

I have had many issues with odd CI failures - that get resolved by one of the two following remedies: A) Either restarting the CI or B) deleting the cache and restarting the CI.

Below is a summary:

I have seen this a bunch of times - timing?

https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/1

1: --------------------------------------------------------------------------------
1:   [hysteresis]                    STARTING TEST
1:   [hysteresis]                    FAIL
1: INFO  [tests] RUNNING TEST: _init_false
1: INFO  [tests] TEST PASSED: _init_false
1: INFO  [tests] RUNNING TEST: _init_true
1: INFO  [tests] TEST PASSED: _init_true

Remedy A) rixes it.

The next class I keep seeing is unresolved externals in travis. Usually related to a lot of builtin command symbols:

sercon serdis

In this case only remedy B) works.

Unfortunately I do not have a capture - but I will post one the next time it happens

julianoes commented 8 years ago

@davids5 It does use usleep under the hood. That's probably just a bad idea for CI. We'll need to mock time at some point for the unit tests.

dagar commented 8 years ago

Alright, I'll take a look at that hysteresis test when I have some time. @LorenzMeier or @julianoes are you seeing intermittent hysteresis test failures anywhere else?

Overall the only way this gets better is by not tolerating intermittent failures and weird CI behaviour. Please be super annoying and add a comment here every time you see a failure like this. One line summary and the build link.

dagar commented 8 years ago

With the posix daemon (https://github.com/PX4/Firmware/pull/5162) stuff it should be possible to split up the tests, which means we can start gathering test reports to isolate particular failures over time. Circleci already has support for this if we convert the ctest output to junit style.

davids5 commented 8 years ago

@dager

This build would not finish

https://circleci.com/gh/PX4/Firmware/1224?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

BFormat checks passed

Error deleting container: Error response from daemon: Driver btrfs failed to remove root filesystem 03f2fe2d46a055406cc9ae319dadf840b3999bdfe2db721ed6b962cd9b8a9a71: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 03f2fe2d46a055406cc9ae319dadf840b3999bdfe2db721ed6b962cd9b8a9a71: operation not permitted

Had to cancel it to get Success!

davids5 commented 8 years ago

@dagar @LorenzMeier

I have an idea that I think will speed up the CI build a lot!

I makes the most sense for CI and not the regular build be cause of the dependency checks for parameters.

If we split the parameter generation into 2 pieces. The first does the complete walk of the tree and builds an xml file of all the parameters and saves it to a parameter.cache.xml.

The second pass uses the parameter.cache.xml. file if it exists and skips that part of the process and on to do the culling and parameter generation for each build based on the modules in the cmake.

dagar commented 8 years ago

That circleci error has something to do with their docker setup. I'll contact them.

I suggest we time how long each phase is actually taking before we try any build optimization. To me it looks like param generation is only a few seconds per build.

davids5 commented 8 years ago

@dagar

Here is an Example of the the phantom errors related to the built ins:

/lib/ccache/arm-none-eabi-g++   -g -fno-exceptions -fno-rtti -std=gnu++0x -fno-threadsafe-statics -DCONFIG_WCHAR_BUILTIN -D__CUSTOM_FILE_IO__ -fcheck-new -Wall -Werror -Wextra -Wno-sign-compare -Wshadow -Wfloat-equal -Wpointer-arith -Wmissing-declarations -Wno-unused-parameter -Werror=format-security -Werror=array-bounds -Wfatal-errors -Werror=unused-variable -Werror=reorder -Werror=uninitialized -Werror=init-self -Werror=unused-but-set-variable -Wformat=1 -Wdouble-promotion -Werror=double-promotion -Wno-missing-field-initializers -fvisibility=hidden -include visibility.h -mcpu=cortex-m4 -mthumb -march=armv7e-m -mfpu=fpv4-sp-d16 -mfloat-abi=hard -nodefaultlibs -nostdlib -Os -fno-strict-aliasing -fomit-frame-pointer -funsafe-math-optimizations -ffunction-sections -fdata-sections -fno-strength-reduce -fno-builtin-printf -O2 -g -DNDEBUG  -Wl,--warn-common -Wl,--gc-sections   -T/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/build/ld.script -Wl,-Map=/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/main.map src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj  -o src/firmware/nuttx/firmware_nuttx -L/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs  -L/home/runner/Firmware/build_mindpx-v2_default/external/Install/lib -Wl,--warn-common -Wl,--gc-sections -Wl,--start-group src/drivers/device/libdrivers__device.a src/drivers/stm32/libdrivers__stm32.a src/drivers/stm32/adc/libdrivers__stm32__adc.a src/drivers/stm32/tone_alarm/libdrivers__stm32__tone_alarm.a src/drivers/led/libdrivers__led.a src/drivers/px4fmu/libdrivers__px4fmu.a src/drivers/boards/mindpx-v2/libdrivers__boards__mindpx-v2.a src/drivers/rgbled_pwm/libdrivers__rgbled_pwm.a src/drivers/mpu6500/libdrivers__mpu6500.a src/drivers/lsm303d/libdrivers__lsm303d.a src/drivers/l3gd20/libdrivers__l3gd20.a src/drivers/hmc5883/libdrivers__hmc5883.a src/drivers/ms5611/libdrivers__ms5611.a src/drivers/srf02_i2c/libdrivers__srf02_i2c.a src/drivers/hc_sr04/libdrivers__hc_sr04.a src/drivers/gps/libdrivers__gps.a src/drivers/pwm_out_sim/libdrivers__pwm_out_sim.a src/drivers/blinkm/libdrivers__blinkm.a src/drivers/airspeed/libdrivers__airspeed.a src/drivers/ets_airspeed/libdrivers__ets_airspeed.a src/drivers/meas_airspeed/libdrivers__meas_airspeed.a src/drivers/frsky_telemetry/libdrivers__frsky_telemetry.a src/modules/sensors/libmodules__sensors.a src/drivers/px4flow/libdrivers__px4flow.a src/drivers/vmount/libdrivers__vmount.a src/drivers/pwm_input/libdrivers__pwm_input.a src/drivers/camera_trigger/libdrivers__camera_trigger.a src/drivers/bst/libdrivers__bst.a src/drivers/snapdragon_rc_pwm/libdrivers__snapdragon_rc_pwm.a src/systemcmds/bl_update/libsystemcmds__bl_update.a src/systemcmds/mixer/libsystemcmds__mixer.a src/systemcmds/param/libsystemcmds__param.a src/systemcmds/perf/libsystemcmds__perf.a src/systemcmds/pwm/libsystemcmds__pwm.a src/systemcmds/esc_calib/libsystemcmds__esc_calib.a src/systemcmds/hardfault_log/libsystemcmds__hardfault_log.a src/systemcmds/reboot/libsystemcmds__reboot.a src/systemcmds/topic_listener/libsystemcmds__topic_listener.a src/systemcmds/top/libsystemcmds__top.a src/systemcmds/config/libsystemcmds__config.a src/systemcmds/nshterm/libsystemcmds__nshterm.a src/systemcmds/mtd/libsystemcmds__mtd.a src/systemcmds/dumpfile/libsystemcmds__dumpfile.a src/systemcmds/ver/libsystemcmds__ver.a src/systemcmds/sd_bench/libsystemcmds__sd_bench.a src/systemcmds/motor_ramp/libsystemcmds__motor_ramp.a src/drivers/sf0x/sf0x_tests/libdrivers__sf0x__sf0x_tests.a src/drivers/test_ppm/libdrivers__test_ppm.a src/modules/commander/commander_tests/libmodules__commander__commander_tests.a src/modules/mc_pos_control/mc_pos_control_tests/libmodules__mc_pos_control__mc_pos_control_tests.a src/modules/controllib_test/libmodules__controllib_test.a src/modules/mavlink/mavlink_tests/libmodules__mavlink__mavlink_tests.a src/modules/unit_test/libmodules__unit_test.a src/modules/uORB/uORB_tests/libmodules__uORB__uORB_tests.a src/systemcmds/tests/libsystemcmds__tests.a src/modules/commander/libmodules__commander.a src/modules/load_mon/libmodules__load_mon.a src/modules/navigator/libmodules__navigator.a src/modules/mavlink/libmodules__mavlink.a src/modules/gpio_led/libmodules__gpio_led.a src/modules/uavcan/libmodules__uavcan.a src/modules/land_detector/libmodules__land_detector.a src/modules/attitude_estimator_q/libmodules__attitude_estimator_q.a src/modules/position_estimator_inav/libmodules__position_estimator_inav.a src/modules/local_position_estimator/libmodules__local_position_estimator.a src/modules/ekf2/libmodules__ekf2.a src/modules/fw_pos_control_l1/libmodules__fw_pos_control_l1.a src/modules/fw_att_control/libmodules__fw_att_control.a src/modules/mc_att_control/libmodules__mc_att_control.a src/modules/mc_pos_control/libmodules__mc_pos_control.a src/modules/vtol_att_control/libmodules__vtol_att_control.a src/modules/sdlog2/libmodules__sdlog2.a src/modules/logger/libmodules__logger.a src/modules/param/libmodules__param.a src/modules/systemlib/libmodules__systemlib.a src/modules/systemlib/mixer/libmodules__systemlib__mixer.a src/modules/uORB/libmodules__uORB.a src/modules/dataman/libmodules__dataman.a src/lib/controllib/liblib__controllib.a src/lib/mathlib/liblib__mathlib.a src/lib/mathlib/math/filter/liblib__mathlib__math__filter.a src/lib/ecl/liblib__ecl.a src/lib/external_lgpl/liblib__external_lgpl.a src/lib/geo/liblib__geo.a src/lib/geo_lookup/liblib__geo_lookup.a src/lib/conversion/liblib__conversion.a src/lib/launchdetection/liblib__launchdetection.a src/lib/terrain_estimation/liblib__terrain_estimation.a src/lib/runway_takeoff/liblib__runway_takeoff.a src/lib/tailsitter_recovery/liblib__tailsitter_recovery.a src/lib/DriverFramework/framework/liblib__DriverFramework__framework.a src/platforms/nuttx/libplatforms__nuttx.a src/platforms/common/libplatforms__common.a src/platforms/nuttx/px4_layer/libplatforms__nuttx__px4_layer.a src/modules/uavcan/libuavcan/libuavcan/libuavcan.a src/modules/uavcan/libuavcan/libuavcan_drivers/stm32/driver/libuavcan_stm32_driver.a -lapps -lnuttx -lm -lgcc src/firmware/nuttx/libromfs.a -Wl,--end-group libmsg_gen.a src/lib/DriverFramework/framework/src/libdf_driver_framework.a && :
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs/libapps.a(nsh_builtin.o): In function `nsh_builtin':
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/apps/nshlib/nsh_builtin.c:105: undefined reference to `exec_builtin'
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs/libnuttx.a(binfmt-libbuiltin_getname.o): In function `builtin_getname':
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx/binfmt/libbuiltin/libbuiltin_getname.c:88: undefined reference to `builtin_for_index'
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx-export/libs/libnuttx.a(binfmt-builtin.o): In function `builtin_loadbinary':
/home/runner/Firmware/build_mindpx-v2_default/mindpx-v2/NuttX/nuttx/binfmt/builtin.c:141: undefined reference to `builtin_for_index'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Makefile:166: recipe for target 'mindpx-v2_default' failed
make[1]: *** [mindpx-v2_default] Error 1
Makefile:278: recipe for target 'check_mindpx-v2_default' failed
make: *** [check_mindpx-v2_default] Error 2

and https://travis-ci.org/PX4/Firmware/jobs/170321947

Building px4nucleoF767ZI-v1_default
GIT_SUBMODULES_ARE_EVIL is defined - Skipping All submodule checking!
-- nuttx-px4nucleoF767ZI-v1-default
-- CMAKE_INSTALL_PREFIX: 
-- cmake module path: /home/travis/build/PX4/Firmware/cmake
-- Nuttx build for px4nucleoF767ZI-v1 on m7 hardware, using nsh with ROMFS on px4fmu_common
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib/ccache/arm-none-eabi-gcc
-- catkin DISABLED
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7") 
-- NuttX patch: nuttx-patches/Fixed-Shadow-wanings.patch
-- NuttX patch: nuttx-patches/nsh_parse.patch
-- NuttX patch: nuttx-patches/c++11.patch
-- NuttX patch: nuttx-patches/wip_inflight_to_upstream.patch
-- NuttX patch: nuttx-patches/silence-jobserver-warnings.patch
-- NuttX patch: nuttx-patches/static_asset-fix.patch
-- NuttX patch: nuttx-patches/ctype_cctype_.patch
-- NuttX patch: nuttx-patches/fixStdint.patch
-- NuttX patch: nuttx-patches/math.h.patch
-- Using C++03
-- Release build type: RelWithDebInfo
-- Adding UAVCAN STM32 platform driver
-- Adding ROMFS on px4nucleoF767ZI-v1
-- GIT_TAG = v1.5.0-1959-gf5b9cdb
-- Configuring done
-- Generating done
-- Build files have been written to: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
PX4 CONFIG: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
ninja: Entering directory `/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default'
[542/543] Linking CXX executable src/firmware/nuttx/firmware_nuttx
FAILED: : && /usr/lib/ccache/arm-none-eabi-g++   -g -fno-exceptions -fno-rtti -std=gnu++0x -fno-threadsafe-statics -DCONFIG_WCHAR_BUILTIN -D__CUSTOM_FILE_IO__ -fcheck-new -Wall -Werror -Wextra -Wno-sign-compare -Wshadow -Wfloat-equal -Wpointer-arith -Wmissing-declarations -Wno-unused-parameter -Werror=format-security -Werror=array-bounds -Wfatal-errors -Werror=unused-variable -Werror=reorder -Werror=uninitialized -Werror=init-self -Werror=unused-but-set-variable -Wformat=1 -Wdouble-promotion -Werror=double-promotion -Wno-missing-field-initializers -fvisibility=hidden -include visibility.h -mcpu=cortex-m7 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=hard -nodefaultlibs -nostdlib -Os -fno-strict-aliasing -fomit-frame-pointer -funsafe-math-optimizations -ffunction-sections -fdata-sections -fno-strength-reduce -fno-builtin-printf -O2 -g -DNDEBUG  -Wl,--warn-common -Wl,--gc-sections   -T/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/build/ld.script -Wl,-Map=/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/main.map src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj  -o src/firmware/nuttx/firmware_nuttx -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/libs  -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/external/Install/lib -Wl,--warn-common -Wl,--gc-sections -Wl,--start-group src/drivers/device/libdrivers__device.a src/drivers/stm32/libdrivers__stm32.a src/drivers/stm32/adc/libdrivers__stm32__adc.a src/drivers/stm32/tone_alarm/libdrivers__stm32__tone_alarm.a src/drivers/led/libdrivers__led.a src/drivers/px4fmu/libdrivers__px4fmu.a src/drivers/boards/px4nucleoF767ZI-v1/libdrivers__boards__px4nucleoF767ZI-v1.a src/drivers/rgbled/libdrivers__rgbled.a src/drivers/mpu6000/libdrivers__mpu6000.a src/drivers/mpu9250/libdrivers__mpu9250.a src/drivers/hmc5883/libdrivers__hmc5883.a src/drivers/ms5611/libdrivers__ms5611.a src/drivers/mb12xx/libdrivers__mb12xx.a src/drivers/srf02/libdrivers__srf02.a src/drivers/sf0x/libdrivers__sf0x.a src/drivers/ll40ls/libdrivers__ll40ls.a src/drivers/trone/libdrivers__trone.a src/drivers/gps/libdrivers__gps.a src/drivers/pwm_out_sim/libdrivers__pwm_out_sim.a src/drivers/hott/libdrivers__hott.a src/drivers/hott/hott_telemetry/libdrivers__hott__hott_telemetry.a src/drivers/hott/hott_sensors/libdrivers__hott__hott_sensors.a src/drivers/blinkm/libdrivers__blinkm.a src/drivers/airspeed/libdrivers__airspeed.a src/drivers/ets_airspeed/libdrivers__ets_airspeed.a src/drivers/meas_airspeed/libdrivers__meas_airspeed.a src/drivers/frsky_telemetry/libdrivers__frsky_telemetry.a src/modules/sensors/libmodules__sensors.a src/drivers/mkblctrl/libdrivers__mkblctrl.a src/drivers/px4flow/libdrivers__px4flow.a src/drivers/oreoled/libdrivers__oreoled.a src/drivers/pwm_input/libdrivers__pwm_input.a src/drivers/camera_trigger/libdrivers__camera_trigger.a src/drivers/bst/libdrivers__bst.a src/drivers/snapdragon_rc_pwm/libdrivers__snapdragon_rc_pwm.a src/drivers/lis3mdl/libdrivers__lis3mdl.a src/drivers/bmp280/libdrivers__bmp280.a src/drivers/bma180/libdrivers__bma180.a src/drivers/bmi160/libdrivers__bmi160.a src/drivers/tap_esc/libdrivers__tap_esc.a src/systemcmds/bl_update/libsystemcmds__bl_update.a src/systemcmds/config/libsystemcmds__config.a src/systemcmds/dumpfile/libsystemcmds__dumpfile.a src/systemcmds/esc_calib/libsystemcmds__esc_calib.a src/systemcmds/hardfault_log/libsystemcmds__hardfault_log.a src/systemcmds/mixer/libsystemcmds__mixer.a src/systemcmds/motor_ramp/libsystemcmds__motor_ramp.a src/systemcmds/mtd/libsystemcmds__mtd.a src/systemcmds/nshterm/libsystemcmds__nshterm.a src/systemcmds/param/libsystemcmds__param.a src/systemcmds/perf/libsystemcmds__perf.a src/systemcmds/pwm/libsystemcmds__pwm.a src/systemcmds/reboot/libsystemcmds__reboot.a src/systemcmds/sd_bench/libsystemcmds__sd_bench.a src/systemcmds/top/libsystemcmds__top.a src/systemcmds/topic_listener/libsystemcmds__topic_listener.a src/systemcmds/ver/libsystemcmds__ver.a src/modules/commander/libmodules__commander.a src/modules/load_mon/libmodules__load_mon.a src/modules/navigator/libmodules__navigator.a src/modules/mavlink/libmodules__mavlink.a src/modules/gpio_led/libmodules__gpio_led.a src/modules/uavcan/libmodules__uavcan.a src/modules/land_detector/libmodules__land_detector.a src/modules/attitude_estimator_q/libmodules__attitude_estimator_q.a src/modules/position_estimator_inav/libmodules__position_estimator_inav.a src/modules/local_position_estimator/libmodules__local_position_estimator.a src/modules/ekf2/libmodules__ekf2.a src/modules/fw_pos_control_l1/libmodules__fw_pos_control_l1.a src/modules/fw_att_control/libmodules__fw_att_control.a src/modules/mc_att_control/libmodules__mc_att_control.a src/modules/mc_pos_control/libmodules__mc_pos_control.a src/modules/vtol_att_control/libmodules__vtol_att_control.a src/modules/logger/libmodules__logger.a src/modules/sdlog2/libmodules__sdlog2.a src/modules/param/libmodules__param.a src/modules/systemlib/libmodules__systemlib.a src/modules/systemlib/mixer/libmodules__systemlib__mixer.a src/modules/uORB/libmodules__uORB.a src/modules/dataman/libmodules__dataman.a src/lib/controllib/liblib__controllib.a src/lib/mathlib/liblib__mathlib.a src/lib/mathlib/math/filter/liblib__mathlib__math__filter.a src/lib/rc/liblib__rc.a src/lib/ecl/liblib__ecl.a src/lib/external_lgpl/liblib__external_lgpl.a src/lib/geo/liblib__geo.a src/lib/geo_lookup/liblib__geo_lookup.a src/lib/conversion/liblib__conversion.a src/lib/launchdetection/liblib__launchdetection.a src/lib/terrain_estimation/liblib__terrain_estimation.a src/lib/runway_takeoff/liblib__runway_takeoff.a src/lib/tailsitter_recovery/liblib__tailsitter_recovery.a src/lib/DriverFramework/framework/liblib__DriverFramework__framework.a src/platforms/nuttx/libplatforms__nuttx.a src/platforms/common/libplatforms__common.a src/platforms/nuttx/px4_layer/libplatforms__nuttx__px4_layer.a src/modules/bottle_drop/libmodules__bottle_drop.a src/examples/rover_steering_control/libexamples__rover_steering_control.a src/examples/px4_simple_app/libexamples__px4_simple_app.a src/modules/uavcan/libuavcan/libuavcan/libuavcan.a src/modules/uavcan/libuavcan/libuavcan_drivers/stm32/driver/libuavcan_stm32_driver.a -lapps -lnuttx -lm -lgcc src/firmware/nuttx/libromfs.a -Wl,--end-group libmsg_gen.a src/lib/DriverFramework/framework/src/libdf_driver_framework.a && :
src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj:(.rodata.g_builtins+0x4ac): undefined reference to `serdis_main'
src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj:(.rodata.g_builtins+0x4bc): undefined reference to `sercon_main'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Makefile:166: recipe for target 'px4nucleoF767ZI-v1_default' failed
make[1]: *** [px4nucleoF767ZI-v1_default] Error 1
Makefile:278: recipe for target 'check_px4nucleoF767ZI-v1_default' failed
make: *** [check_px4nucleoF767ZI-v1_default] Error 2
The command "if [ "${TRAVIS_OS_NAME}" = "linux" ]; then docker run --rm -v `pwd`:`pwd`:rw -v $HOME/.ccache:$HOME/.ccache:rw -e CCACHE_DIR=$HOME/.ccache -e CI=true -e GIT_SUBMODULES_ARE_EVIL=1 -w=`pwd` --user=$UID -it ${DOCKER_REPO} /bin/bash -c "make check VECTORCONTROL=1"; elif [ "${TRAVIS_OS_NAME}" = "osx" ]; then make tests; fi" exited with 2.
dagar commented 8 years ago

Are you mainly seeing these on travis-ci, semaphore or locally? I feel like if it was happening in master someone would have mentioned it by now. I should take a look at your nuttx_v3 cmake changes.

davids5 commented 8 years ago

@dagar

Only on CI.

Please do!

davids5 commented 8 years ago

@dagar

This was the "Need more boxes" I had mentioned

image

julianoes commented 8 years ago

I'm not following here but if the problem is only the usleeps in the hysteresis check, let's disable it until we have mocked time.

dagar commented 8 years ago

@julianoes this has become a bit of a catch all for CI failures that @davids5's been hitting.

davids5 commented 8 years ago

@dagar more of the same

Building px4nucleoF767ZI-v1_default
GIT_SUBMODULES_ARE_EVIL is defined - Skipping All submodule checking!
-- nuttx-px4nucleoF767ZI-v1-default
-- CMAKE_INSTALL_PREFIX: 
-- cmake module path: /home/travis/build/PX4/Firmware/cmake
-- Nuttx build for px4nucleoF767ZI-v1 on m7 hardware, using nsh with ROMFS on px4fmu_common
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib/ccache/arm-none-eabi-gcc
-- catkin DISABLED
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7") 
-- NuttX patch: nuttx-patches/Fixed-Shadow-wanings.patch
-- NuttX patch: nuttx-patches/nsh_parse.patch
-- NuttX patch: nuttx-patches/c++11.patch
-- NuttX patch: nuttx-patches/wip_inflight_to_upstream.patch
-- NuttX patch: nuttx-patches/silence-jobserver-warnings.patch
-- NuttX patch: nuttx-patches/static_asset-fix.patch
-- NuttX patch: nuttx-patches/wip_nxphlite_nuttx_cfg.patch
-- NuttX patch: nuttx-patches/ctype_cctype_.patch
-- NuttX patch: nuttx-patches/fixStdint.patch
-- NuttX patch: nuttx-patches/math.h.patch
-- Using C++03
-- Release build type: RelWithDebInfo
-- Adding UAVCAN STM32 platform driver
-- Adding ROMFS on px4nucleoF767ZI-v1
-- GIT_TAG = v1.5.0-1979-g866ec5f
-- Configuring done
-- Generating done
-- Build files have been written to: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
PX4 CONFIG: /home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default
ninja: Entering directory `/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default'
[543/544] Linking CXX executable src/firmware/nuttx/firmware_nuttx
FAILED: : && /usr/lib/ccache/arm-none-eabi-g++   -g -fno-exceptions -fno-rtti -std=gnu++0x -fno-threadsafe-statics -DCONFIG_WCHAR_BUILTIN -D__CUSTOM_FILE_IO__ -fcheck-new -Wall -Werror -Wextra -Wno-sign-compare -Wshadow -Wfloat-equal -Wpointer-arith -Wmissing-declarations -Wno-unused-parameter -Werror=format-security -Werror=array-bounds -Wfatal-errors -Werror=unused-variable -Werror=reorder -Werror=uninitialized -Werror=init-self -Werror=unused-but-set-variable -Wformat=1 -Wdouble-promotion -Werror=double-promotion -Wno-missing-field-initializers -fvisibility=hidden -include visibility.h -mcpu=cortex-m7 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=hard -nodefaultlibs -nostdlib -Os -fno-strict-aliasing -fomit-frame-pointer -funsafe-math-optimizations -ffunction-sections -fdata-sections -fno-strength-reduce -fno-builtin-printf -O2 -g -DNDEBUG  -Wl,--warn-common -Wl,--gc-sections   -T/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/build/ld.script -Wl,-Map=/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/main.map src/firmware/nuttx/CMakeFiles/firmware_nuttx.dir/builtin_commands.c.obj  -o src/firmware/nuttx/firmware_nuttx -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/libs  -L/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/external/Install/lib -Wl,--warn-common -Wl,--gc-sections -Wl,--start-group src/drivers/device/libdrivers__device.a src/drivers/stm32/libdrivers__stm32.a src/drivers/stm32/adc/libdrivers__stm32__adc.a src/drivers/stm32/tone_alarm/libdrivers__stm32__tone_alarm.a src/drivers/led/libdrivers__led.a src/drivers/px4fmu/libdrivers__px4fmu.a src/drivers/boards/px4nucleoF767ZI-v1/libdrivers__boards__px4nucleoF767ZI-v1.a src/drivers/rgbled/libdrivers__rgbled.a src/drivers/mpu6000/libdrivers__mpu6000.a src/drivers/mpu9250/libdrivers__mpu9250.a src/drivers/hmc5883/libdrivers__hmc5883.a src/drivers/ms5611/libdrivers__ms5611.a src/drivers/mb12xx/libdrivers__mb12xx.a src/drivers/srf02/libdrivers__srf02.a src/drivers/sf0x/libdrivers__sf0x.a src/drivers/ll40ls/libdrivers__ll40ls.a src/drivers/trone/libdrivers__trone.a src/drivers/gps/libdrivers__gps.a src/drivers/pwm_out_sim/libdrivers__pwm_out_sim.a src/drivers/hott/libdrivers__hott.a src/drivers/hott/hott_telemetry/libdrivers__hott__hott_telemetry.a src/drivers/hott/hott_sensors/libdrivers__hott__hott_sensors.a src/drivers/blinkm/libdrivers__blinkm.a src/drivers/airspeed/libdrivers__airspeed.a src/drivers/ets_airspeed/libdrivers__ets_airspeed.a src/drivers/meas_airspeed/libdrivers__meas_airspeed.a src/drivers/frsky_telemetry/libdrivers__frsky_telemetry.a src/modules/sensors/libmodules__sensors.a src/drivers/mkblctrl/libdrivers__mkblctrl.a src/drivers/px4flow/libdrivers__px4flow.a src/drivers/oreoled/libdrivers__oreoled.a src/drivers/pwm_input/libdrivers__pwm_input.a src/drivers/camera_trigger/libdrivers__camera_trigger.a src/drivers/bst/libdrivers__bst.a src/drivers/snapdragon_rc_pwm/libdrivers__snapdragon_rc_pwm.a src/drivers/lis3mdl/libdrivers__lis3mdl.a src/drivers/bmp280/libdrivers__bmp280.a src/drivers/bma180/libdrivers__bma180.a src/drivers/bmi160/libdrivers__bmi160.a src/drivers/tap_esc/libdrivers__tap_esc.a src/systemcmds/bl_update/libsystemcmds__bl_update.a src/systemcmds/config/libsystemcmds__config.a src/systemcmds/dumpfile/libsystemcmds__dumpfile.a src/systemcmds/esc_calib/libsystemcmds__esc_calib.a src/systemcmds/hardfault_log/libsystemcmds__hardfault_log.a src/systemcmds/mixer/libsystemcmds__mixer.a src/systemcmds/motor_ramp/libsystemcmds__motor_ramp.a src/systemcmds/mtd/libsystemcmds__mtd.a src/systemcmds/nshterm/libsystemcmds__nshterm.a src/systemcmds/param/libsystemcmds__param.a src/systemcmds/perf/libsystemcmds__perf.a src/systemcmds/pwm/libsystemcmds__pwm.a src/systemcmds/reboot/libsystemcmds__reboot.a src/systemcmds/sd_bench/libsystemcmds__sd_bench.a src/systemcmds/top/libsystemcmds__top.a src/systemcmds/topic_listener/libsystemcmds__topic_listener.a src/systemcmds/ver/libsystemcmds__ver.a src/modules/commander/libmodules__commander.a src/modules/load_mon/libmodules__load_mon.a src/modules/navigator/libmodules__navigator.a src/modules/mavlink/libmodules__mavlink.a src/modules/gpio_led/libmodules__gpio_led.a src/modules/uavcan/libmodules__uavcan.a src/modules/land_detector/libmodules__land_detector.a src/modules/attitude_estimator_q/libmodules__attitude_estimator_q.a src/modules/position_estimator_inav/libmodules__position_estimator_inav.a src/modules/local_position_estimator/libmodules__local_position_estimator.a src/modules/ekf2/libmodules__ekf2.a src/modules/fw_pos_control_l1/libmodules__fw_pos_control_l1.a src/modules/fw_att_control/libmodules__fw_att_control.a src/modules/mc_att_control/libmodules__mc_att_control.a src/modules/mc_pos_control/libmodules__mc_pos_control.a src/modules/vtol_att_control/libmodules__vtol_att_control.a src/modules/logger/libmodules__logger.a src/modules/sdlog2/libmodules__sdlog2.a src/modules/param/libmodules__param.a src/modules/systemlib/libmodules__systemlib.a src/modules/systemlib/mixer/libmodules__systemlib__mixer.a src/modules/uORB/libmodules__uORB.a src/modules/dataman/libmodules__dataman.a src/lib/controllib/liblib__controllib.a src/lib/mathlib/liblib__mathlib.a src/lib/mathlib/math/filter/liblib__mathlib__math__filter.a src/lib/rc/liblib__rc.a src/lib/ecl/liblib__ecl.a src/lib/external_lgpl/liblib__external_lgpl.a src/lib/geo/liblib__geo.a src/lib/geo_lookup/liblib__geo_lookup.a src/lib/conversion/liblib__conversion.a src/lib/launchdetection/liblib__launchdetection.a src/lib/terrain_estimation/liblib__terrain_estimation.a src/lib/runway_takeoff/liblib__runway_takeoff.a src/lib/tailsitter_recovery/liblib__tailsitter_recovery.a src/lib/DriverFramework/framework/liblib__DriverFramework__framework.a src/platforms/nuttx/libplatforms__nuttx.a src/platforms/common/libplatforms__common.a src/platforms/nuttx/px4_layer/libplatforms__nuttx__px4_layer.a src/modules/bottle_drop/libmodules__bottle_drop.a src/examples/rover_steering_control/libexamples__rover_steering_control.a src/examples/px4_simple_app/libexamples__px4_simple_app.a src/modules/uavcan/libuavcan/libuavcan/libuavcan.a src/modules/uavcan/libuavcan/libuavcan_drivers/stm32/driver/libuavcan_stm32_driver.a -lapps -lnuttx -lm -lgcc src/firmware/nuttx/libromfs.a -Wl,--end-group libmsg_gen.a src/lib/DriverFramework/framework/src/libdf_driver_framework.a && :
/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/nuttx-export/libs/libapps.a(nsh_session.o): In function `nsh_session':
/home/travis/build/PX4/Firmware/build_px4nucleoF767ZI-v1_default/px4nucleoF767ZI-v1/NuttX/apps/nshlib/nsh_session.c:148: undefined reference to `readline'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Makefile:166: recipe for target 'px4nucleoF767ZI-v1_default' failed
make[1]: *** [px4nucleoF767ZI-v1_default] Error 1
Makefile:279: recipe for target 'check_px4nucleoF767ZI-v1_default' failed
make: *** [check_px4nucleoF767ZI-v1_default] Error 2
davids5 commented 8 years ago

@dagar - sigh https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/12

dagar commented 8 years ago

It looks like our environment + builds don't leave enough room to compress the docker image. I've removed the docker snapshot from semaphoreci for now. It didn't save that much time.

davids5 commented 8 years ago

@dagar more [hysteresis] FAIL

dagar commented 8 years ago

Was that a different build? I don't see that failure.

davids5 commented 8 years ago

I restarted it

dagar commented 8 years ago

Ah okay, double fail.

davids5 commented 8 years ago

This one is || build issue https://circleci.com/gh/PX4/Firmware/1657?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

davids5 commented 8 years ago

Same here https://semaphoreci.com/px4/firmware/branches/nuttx_v3_fmuv5/builds/1

davids5 commented 8 years ago

@LorenzMeier said we are going to switch over to nuttx_v3 in the next 3 weeks or so. So we have better get the demons out.

dagar commented 8 years ago

Got it. I'll have a little bit of time to work through these issues later tonight.

davids5 commented 8 years ago

@dagar thank you!

davids5 commented 8 years ago

@dagar

https://travis-ci.org/PX4/Firmware/jobs/178478801

I knew I should have never dated Travis's sister :)

dagar commented 8 years ago

That's a somewhat regular failure where travis-ci can't reach github via http. Might be more reliable over ssh, but then it's slightly harder for people to access the repo.

davids5 commented 7 years ago

@dagar

Run out of space:

https://semaphoreci.com/px4/firmware/branches/nuttx_v3_v4_pro/builds/2

davids5 commented 7 years ago

@dagar We are going to need 

arm-none-eabi-gcc (GNU Tools for ARM Embedded Processors) 5.4.1 20160609 (release) [ARM/embedded-5-branch revision 237715]
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

on semaphoreci.com

https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/16

Is there a way to install both 9.3 and 5.4.1 compilers and then have the make file or a script select it?

dagar commented 7 years ago

In this particular case for semaphoreci we can move it up to 5.4.1 by pointing it at a different docker image. GCC 4.9 stable builds for QGC are handled on travis-ci.

We could have larger images with both installed, but I'm not sure if there's a good way for the build system to select a version, and in fact it should probably always respect what the user/developer sets for CC and CXX.

If we have minimum version requirements can you enforce them in cmake? Perhaps here - https://github.com/PX4/Firmware/blob/master/cmake/nuttx/px4_impl_nuttx.cmake

davids5 commented 7 years ago

@dagar please rev semaphoreci to 5.41 then.

Also Please do the OR for nuttx_v3 that does the enforcement. Should we have an override from the env to help with A/B testing?

davids5 commented 7 years ago

@dagar

How can we fix this.

https://semaphoreci.com/px4/firmware/branches/nuttx_v3_fmuv5/builds/4

rsync: recv_generator: mkdir "/home/runner/Firmware/build_px4esc-v1_bootloader/px4esc-v1/NuttX/nuttx/wireless" failed: No space left on device (28)

dagar commented 7 years ago

I removed docker image caching from semaphoreci and wiped the existing cache. It's not clear how much space they give us per build. I restarted your build.

davids5 commented 7 years ago

@dagar - still no joy!

https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/22

dagar commented 7 years ago

I only partially disabled ccache before. Trying again with it completely disabled and wiped. Build restarted - https://semaphoreci.com/px4/firmware/branches/nuttx_v3/builds/22

davids5 commented 7 years ago

@dagar - this is starting to remind me of a movie with zombies that can not be killed.

Fatal error: can't create at45db.o: No space left on device
dagar commented 7 years ago

I guess we're just running out of space with all the individual builds? Each has a copy of nuttx, build artifacts, etc. Can you add df -h between each build or something?

davids5 commented 7 years ago

@dagar

I nodded you expertise - I think that last parallelism change we made for CI broke a non nija build

  3%] Copying NuttX for px4fmu-v2 with nsh
[  3%] Copying NuttX for px4fmu-v2 with nsh
[  3%] Copying NuttX for px4fmu-v2 with nsh
[  3%] Built target xml_gen
Scanning dependencies of target nuttx_patch_ctype_cctype_.patch-px4fmu-v2
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" -> "NxWidgets/ChangeLog.txt": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" -> "apps/system/zmodem/zm_state.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" -> "misc/tools/kconfig-frontends/scripts/.autostuff/m4/libtool.m4": No such file or directory (2)
[  3%] Copying NuttX for px4fmu-v2 with nsh
file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_ohciram.h.sE3Jqr"
file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_pwm.c.WZr2hd"
IO error encountered -- skipping file deletion
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" -> "nuttx/arch/arm/src/lpc17xx/lpc17_serial.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" -> "nuttx/arch/z16/src/z16f/chip.h": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" -> "nuttx/binfmt/binfmt.h": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" -> "nuttx/audio/audio.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" -> "nuttx/configs/lm4f120-launchpad/src/lm4f_autoleds.c": No such file or directory (2)
rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" failed: No such file or directory (2)
rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" -> "nuttx/conf

How to break it

make clean && make NO_NINJA_BUILD=1 px4fmu-v2_default

How to fix it

make clean && make px4fmu-v2_default

dagar commented 7 years ago

Strange, I wonder if there's an assumption made about a relative path that's actually only valid for the generated ninja, but not make. A bit of a wild guess... I'll definitely find a way to fix this, but likely won't be able to until next week (currently travelling).

On Fri, Dec 16, 2016 at 2:36 PM, David Sidrane notifications@github.com wrote:

@dagar https://github.com/dagar

I nodded you expertise - I think that last parallelism change we made for CI broke a non nija build

3%] Copying NuttX for px4fmu-v2 with nsh [ 3%] Copying NuttX for px4fmu-v2 with nsh [ 3%] Copying NuttX for px4fmu-v2 with nsh [ 3%] Built target xml_gen Scanning dependencies of target nuttx_patch_ctypecctype.patch-px4fmu-v2 rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/NxWidgets/.ChangeLog.txt.ChNkNF" -> "NxWidgets/ChangeLog.txt": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/apps/system/zmodem/.zm_state.c.TcoIPD" -> "apps/system/zmodem/zm_state.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/misc/tools/kconfig-frontends/scripts/.autostuff/m4/.libtool.m4.XwApMP" -> "misc/tools/kconfig-frontends/scripts/.autostuff/m4/libtool.m4": No such file or directory (2) [ 3%] Copying NuttX for px4fmu-v2 with nsh file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_ohciram.h.sE3Jqr" file has vanished: "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_pwm.c.WZr2hd" IO error encountered -- skipping file deletion rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/arm/src/lpc17xx/.lpc17_serial.c.jYmVBT" -> "nuttx/arch/arm/src/lpc17xx/lpc17_serial.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/arch/z16/src/z16f/.chip.h.qaPMCM" -> "nuttx/arch/z16/src/z16f/chip.h": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/binfmt/.binfmt.h.UFvrzA" -> "nuttx/binfmt/binfmt.h": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/audio/.audio.c.ha60K5" -> "nuttx/audio/audio.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lm4f120-launchpad/src/.lm4f_autoleds.c.b84Hv4" -> "nuttx/configs/lm4f120-launchpad/src/lm4f_autoleds.c": No such file or directory (2) rsync: stat "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" failed: No such file or directory (2) rsync: rename "Firmware/build_px4fmu-v2_default/px4fmu-v2/NuttX/nuttx/configs/lpc4370-link2/.README.txt.pIjGiy" -> "nuttx/conf

How to break it

make clean && make NO_NINJA_BUILD=1 px4fmu-v2_default

How to fix it

make clean && make px4fmu-v2_default

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/PX4/Firmware/issues/5717#issuecomment-267677937, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFK6KjRNxz6en4j7_rZ3_NzHDdwOyV3ks5rIuhDgaJpZM4KdWjv .

dagar commented 7 years ago

The cmake generated makefile is actually running the copy multiple times.

dagar commented 7 years ago

@davids5 should be resolved on master. Can you verify?