FMUv2 Nuttx build regression

julianoes commented 5 years ago

I'm sometimes getting a build error when doing a clean build. I started looking into it but it's not obvious to me what happens. I've been able to reproduce it on Ubuntu 16.04 and Fedora 28 both times without ninja-build.

FYI: @bkueng @dagar

CCACHE_DISABLE=1 make px4_fmu-v2_default
-- PX4 config file: /home/julianoes/src/Firmware/boards/px4/fmu-v2/default.cmake
-- PX4 config: px4_fmu-v2_default
-- PX4 platform: nuttx
-- PX4 version: v1.8.0-1912-gdc49e259b3
-- cmake build type: MinSizeRel
-- The CXX compiler identification is GNU 7.2.1
-- The C compiler identification is GNU 7.2.1
-- The ASM compiler identification is GNU
-- Found assembler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-gcc
-- Check for working CXX compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-g++
-- Check for working CXX compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-gcc
-- Check for working C compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /usr/bin/python (found version "2.7.15") 
-- Found PY_jinja2: /usr/lib/python2.7/site-packages/jinja2  
-- PX4 ECL: Very lightweight Estimation & Control Library v0.9.0-608-gacde4eb
-- Building and including px4_io-v2_default
-- ROMFS: px4fmu_common
-- Configuring done
-- Generating done
-- Build files have been written to: /home/julianoes/src/Firmware/build/px4_fmu-v2_default
Scanning dependencies of target git_nuttx_apps
Scanning dependencies of target px4io_firmware
Scanning dependencies of target git_nuttx
Scanning dependencies of target uorb_headers
[  0%] git submodule platforms/nuttx/NuttX/nuttx
[  0%] Creating directories for 'px4io_firmware'
[  0%] git submodule platforms/nuttx/NuttX/apps
[  0%] Generating uORB topic headers
[  0%] Built target git_nuttx
[  0%] Built target git_nuttx_apps
Scanning dependencies of target mixer_gen
Scanning dependencies of target mixer_gen_6dof
[  0%] Generating mixer_multirotor_6dof.generated.h
[  0%] Generating mixer_multirotor.generated.h
[  0%] No download step for 'px4io_firmware'
[  0%] No patch step for 'px4io_firmware'
[  1%] No update step for 'px4io_firmware'
[  1%] Generating mixer_multirotor_normalized.generated.h
[  1%] Performing configure step for 'px4io_firmware'
[  1%] Built target mixer_gen_6dof
Scanning dependencies of target ver_gen
-- PX4 config file: /home/julianoes/src/Firmware/boards/px4/io-v2/default.cmake
[  1%] Generating git version header
-- PX4 config: px4_io-v2_default
-- PX4 platform: nuttx
-- PX4 version: v1.8.0-1912-gdc49e259b3
-- cmake build type: MinSizeRel
[  1%] Built target mixer_gen
-- The CXX compiler identification is GNU 7.2.1
Scanning dependencies of target git_gps_devices
[  1%] git submodule src/drivers/gps/devices
-- The C compiler identification is GNU 7.2.1
-- The ASM compiler identification is GNU
-- Found assembler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-gcc
[  1%] Built target git_gps_devices
-- Check for working CXX compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-g++
Scanning dependencies of target git_ecl
[  1%] git submodule src/lib/ecl
[  1%] Built target ver_gen
Scanning dependencies of target git_mavlink_v2
[  1%] Built target git_ecl
-- Check for working CXX compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-g++ -- works
-- Detecting CXX compiler ABI info
[  2%] git submodule mavlink/include/mavlink/v2.0
[  2%] Built target git_mavlink_v2
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
Scanning dependencies of target nuttx_apps_build
-- Detecting CXX compile features - done
-- Check for working C compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-gcc
Scanning dependencies of target nuttx_configs_build
-- Check for working C compiler: /home/julianoes/Software/gcc-arm-none-eabi-7-2017-q4-major/bin/arm-none-eabi-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
[  2%] Copying NuttX/apps to /home/julianoes/src/Firmware/build/px4_fmu-v2_default/NuttX
-- Found PythonInterp: /usr/bin/python (found version "2.7.15") 
[  2%] Copying NuttX/apps to /home/julianoes/src/Firmware/build/px4_fmu-v2_default/NuttX
-- Found PY_jinja2: /usr/lib/python2.7/site-packages/jinja2  
-- PX4 ECL: Very lightweight Estimation & Control Library v0.9.0-608-gacde4eb
[  2%] Copying NuttX/nuttx to /home/julianoes/src/Firmware/build/px4_fmu-v2_default/NuttX
[  2%] Copying NuttX/nuttx to /home/julianoes/src/Firmware/build/px4_fmu-v2_default/NuttX
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/freedom-k64f/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/freedom-k64f/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/freedom-k66f': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/hymini-stm32v/nsh2': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/hymini-stm32v/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/hymini-stm32v/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/hymini-stm32v/usbserial': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/kwikstik-k40': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lincoln60/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lincoln60/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lincoln60/thttpd-binfs': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lm3s6432-s2e': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lm3s8962-ek/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lm4f120-launchpad': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpc4357-evb/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpc4357-evb/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpc4357-evb/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpc4357-evb/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpc4370-link2': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpcxpresso-lpc1115': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/lpcxpresso-lpc1768': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/micropendous3': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/mikroe-stm32f4': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/mirtoo': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/misoc': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/moteino-mega': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/moxa': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/mx1ads': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nr5m100-nexys4/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nr5m100-nexys4/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nr5m100-nexys4/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/ntosd-dm320': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/adc': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/can': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/hello': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/nxlines': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/pwm': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/serialrx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f303re/uavcan': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-f334r8': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-l432kc': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-l452re': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/nucleo-l476rg': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-lpc-h3131/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-lpc-h3131/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-lpc-h3131/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-lpc-h3131/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-lpc-h3131/tools': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-lpc1766stk': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimex-strp711': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/olimexino-stm32': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/open1788/knsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/open1788/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/open1788/nxlines': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/open1788/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/open1788/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/p112': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/pcblogic-pic32mx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/pcduino-a10/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/pcduino-a10/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/pcduino-a10/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/pcduino-a10/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/photon': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/qemu-i486': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sabre-6quad': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam3u-ek': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4cmp-db': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4e-ek': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4s-xplained-pro/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4s-xplained-pro/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4s-xplained/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4s-xplained/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4s-xplained/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sam4s-xplained/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d2-xult': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/norboot': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/nx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/nxplayer': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/nxwm': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/ov2640': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d3x-ek/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/ipv6': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/knsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/nxwm': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/ramtest': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sama5d4-ek/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/samd20-xplained': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/samd21-xplained': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/same70-xplained': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/bas': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/configdata': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/cxxtest': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/fb': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/ipforward': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/minibasic': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/mount': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/mtdpart': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/mtdrwb': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nettest': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nsh2': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nx11': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nxffs': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nxlines': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/nxwm': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/ostest': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/pashello': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/pf_ieee802154': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/pktradio': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/sixlowpan': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/touchscreen': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/traveler': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/udgram': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/unionfs': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/sim/ustream': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/skp16c26': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/spark': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm3210e-eval': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm3220g-eval': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm3240g-eval': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32f429i-disco': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32f4discovery': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32l476vg-disco/kernel': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32l476vg-disco/knsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32l476vg-disco/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32l476vg-disco/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32l476vg-disco/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/stm32ldiscovery': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/teensy-2.0/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/teensy-2.0/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/teensy-2.0/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/teensy-2.0/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/teensy-2.0/usbmsc': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/teensy-3.x': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/twr-k60n512': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/twr-k64f120m': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/u-blox-c027': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/xmc4500-relax/include': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/xmc4500-relax/nsh': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/xmc4500-relax/scripts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/xmc4500-relax/src': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/xtrs': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/zkit-arm-1769': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/configs/zp214xpa': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/audio': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/bch': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/input': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/leds': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/loop': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/modem': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/net': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/pipes': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/power': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/sensors': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/timers': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/usbdev': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/drivers/usbmisc': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/fs': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/net': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/nx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/power': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/sensors': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/serial': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/spi': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/syslog': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/timers': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/usb': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/video': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/include/nuttx/wireless': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/audio': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/bin': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/dirent': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/dllfcn': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/fixedmath': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/hex2bin': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/inttypes': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/kbin': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/libgen': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/locale': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/machine': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/net': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/netdb': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/pthread': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/stdlib': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/tls': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/ubin': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/unistd': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/wctype': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/wqueue': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libc/zoneinfo': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libnx/nx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libnx/nxfonts': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libnx/nxglib': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libnx/nxmu': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/libxx': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/mm/kmm_heap': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/mm/mm_gran': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/mm/mm_heap': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/mm/shm': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/mm/ubin': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/mm/umm_heap': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/devif': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/icmp': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/icmpv6': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/ieee802154': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/inet': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/ipforward': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/local': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/neighbor': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/netdev': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/pkt': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/procfs': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/route': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/sixlowpan': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/tcp': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/udp': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/usrsock': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/net/utils': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/environ': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/errno': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/group': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/init': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/irq': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/module': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/mqueue': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/semaphore': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/signal': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/task': File exists
make[3]: *** [NuttX/CMakeFiles/nuttx_configs_build.dir/build.make:12610: NuttX/nuttx_copy.stamp] Error 1
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/timer': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/wdog': File exists
cp: cannot create directory 'build/px4_fmu-v2_default/NuttX/nuttx/sched/wqueue': File exists
make[3]: *** [NuttX/CMakeFiles/nuttx_apps_build.dir/build.make:10256: NuttX/nuttx_copy.stamp] Error 1
make[2]: *** [CMakeFiles/Makefile2:8094: NuttX/CMakeFiles/nuttx_configs_build.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/Makefile2:8028: NuttX/CMakeFiles/nuttx_apps_build.dir/all] Error 2

dagar commented 5 years ago

Weird, I'll take a look. I suppose we should get a couple builds on Jenkins that don't use Ninja.

julianoes commented 5 years ago

That would make sense. (Ceterum censeo ninja shouldn't be used.)

julianoes commented 5 years ago

This is another warning/error that I get sometimes:

[  4%] Copying NuttX config nsh and configuring
[  4%] Generating nuttx/include/nuttx/version.h, nuttx/include/nuttx/config.h
make[5]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
/bin/sh: tools/mkversion: Permission denied

I'm not sure if it's related.

julianoes commented 5 years ago

git bisect points somewhere to the board organization:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
abb3817d3121b89e35b6774a4306c86a4616e5cf
cadda8c2630d12fb44cfdc567b68ad251b4f4435
f692ad04d0ea22b8874926e9dc0cac23263bbeae
2d052c02fae3c581a5947f01ace894c54dc1ffbb
We cannot bisect more!

dagar commented 5 years ago

This seems like a race condition in the nuttx part of the build. Almost like it's taking multiple dependency paths to the nuttx setup, but not recognizing that step as a single node and executing it twice.

julianoes commented 5 years ago

Has it always be the case that the Makefile build is different from the ninja build?

$ ls -l | grep bin                                    
-rwxr-xr-x 1 julianoes julianoes 1021685 Dec 19 09:43 px4_fmu-v2.bin.make
-rwxr-xr-x 1 julianoes julianoes 1021613 Dec 19 09:45 px4_fmu-v2.bin.ninja

$ xxd px4_fmu-v2.bin.make > px4_fmu-v2.bin.make.hex
$ xxd px4_fmu-v2.bin.ninja > px4_fmu-v2.bin.ninja.hex
$ diff px4_fmu-v2.bin.make.hex px4_fmu-v2.bin.ninja.hex | wc -l
40323

dagar commented 5 years ago

Is this still relevant?

julianoes commented 5 years ago

Yes, the Makefile build is broken as far as I know, unless it has been fixed.

julianoes commented 5 years ago

I found that the build works with -j1.

julianoes commented 5 years ago

@dagar I'm currently looking at:

/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/nuttx/Make.defs:36: /home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/nuttx/.config: No such file or directory

and I don't understand why we copy the defconfig file twice:

dagar commented 5 years ago

@julianoes the defconfig now only contains non-default values and needs to be reconstituted at cmake configure time (cmake uses the variables).

execute_process does this during configure
custom_command does this at runtime if the dependencies change

julianoes commented 5 years ago

I also found this:

Deleting primary custom command output "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/nuttx/.config" because another output "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/nuttx_config.stamp" does not exist.
Dependee "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/CMakeFiles/nuttx_apps_build.dir/DependInfo.cmake" is newer than depender "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/CMakeFiles/nuttx_apps_build.dir/depend.internal".
Dependee "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/CMakeFiles/nuttx_apps_build.dir/depend.internal".

julianoes commented 5 years ago

And this:

Deleting primary custom command output "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/nuttx/.config" because another output "/home/julianoes/src/Firmware/build/px4_fmu-v4_default/NuttX/nuttx_config.stamp" does not exist.

julianoes commented 5 years ago

Alright, I think this first deletion happens presumably because the file exists from the configure step. It only happens once and at the beginning so I don't think we have to worry about it.

Now, what I found is that the config copy steps are done multiple times if we have -j4 and not -j1, so as you @dagar wrote above there are multiple dependency paths. I have a really hard to time to understand all the dependencies in this file: https://github.com/PX4/Firmware/blob/d9f8e37288d8e34baac79b32d71862045946534b/platforms/nuttx/NuttX/CMakeLists.txt Is there any way we could improve it and make it easier to understand as well as prevent make or cmake from getting confused?

julianoes commented 5 years ago

I believe we're running into this: https://gitlab.kitware.com/cmake/cmake/issues/17585

julianoes commented 5 years ago

I have it compiling again, I'll make a pull request soon.

PX4 / PX4-Autopilot

FMUv2 Nuttx build regression #11003