Issues building MakeCode with CODAL tag v0.2.44

[microbit-carlos]

https://github.com/microsoft/pxt-microbit/actions/runs/3929638137/jobs/6718685276

...
[ 90%] Building CXX object CMakeFiles/MICROBIT.dir/pxtapp/core/images.cpp.o
[ 90%] Building CXX object CMakeFiles/MICROBIT.dir/pxtapp/core/gc.cpp.o
[ 91%] Building CXX object CMakeFiles/MICROBIT.dir/pxtapp/core/input.cpp.o

TypeError: Cannot read property 'length' of null
    at compressHex (/home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:105029:37)
    at /home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:104958:36
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async fillExtInfoAsync (/home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:112337:51)
    at async MainPackage.getCompileOptionsAsync (/home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:112383:33)
    at async /home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:159488:19
    at async /home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:99300:42
    at async Promise.all (index 0)
    at async promisePoolAsync (/home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:99306:21)
    at async buildTargetCoreAsync (/home/runner/work/pxt-microbit/pxt-microbit/node_modules/pxt-core/built/pxt.js:159470:5)

Building with tag v0.2.43 succeeds: https://github.com/microsoft/pxt-microbit/actions/runs/3929577918/jobs/6718550537

Diff between tags: https://github.com/lancaster-university/codal-microbit-v2/compare/v0.2.43...v0.2.44

[microbit-carlos]

Okay, looks like we need to take in consideration the CMake version in the yotta docker images.

$ docker run --rm pext/yotta:latest cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ docker run --rm pext/yotta:update-yotta3 cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

We are currently the using the file(APPEND feature, which was added in 3.12, so we'll have to figure something that'll work in older CMake versions : https://cmake.org/cmake/help/latest/command/file.html#append https://github.com/lancaster-university/codal-microbit-v2/blob/75a0620d9d2391507e7a3e7d0be63ca940285272/CMakeLists.txt#L17

If I install CMake 3.8.2 locally I can replicate the same issue:

$ cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ python build.py
codal-microbit-v2 is already installed
Set target: codal-microbit-v2 
Using target-locked.json
Targeting codal-microbit-v2
-- The C compiler identification is GNU 10.3.1
-- The CXX compiler identification is GNU 10.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM compiler identification is GNU
-- Found assembler: /Users/microbit-carlos/opt/gcc-arm-none-eabi-10.3-2021.10/bin/arm-none-eabi-gcc
Supressing -Wexpansion-to-defined.
-- Adding library path: (/Users/microbit-carlos/workspace/mbef/codal/microbit-v2-samples/libraries)
Installing dependencies...
codal-core is already installed
codal-nrf52 is already installed
codal-microbit-nrf5sdk is already installed
Using library: codal-microbit-v2
Building WITHOUT softdevice support
Using library: codal-core
Using library: codal-nrf52
Using library: codal-microbit-nrf5sdk
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/microbit-carlos/workspace/mbef/codal/microbit-v2-samples/build
Scanning dependencies of target codal-core
[  1%] Building CXX object libraries/codal-core/CMakeFiles/codal-core.dir/source/core/CodalCompat.cpp.obj

...

[ 79%] Building CXX object libraries/codal-microbit-v2/CMakeFiles/codal-microbit-v2.dir/source/MicroBitFileSystem.cpp.obj
In file included from /Users/microbit-carlos/workspace/mbef/codal/microbit-v2-samples/libraries/codal-microbit-v2/model/MicroBit.h:78,
                 from /Users/microbit-carlos/workspace/mbef/codal/microbit-v2-samples/libraries/codal-microbit-v2/source/MicroBitAudio.cpp:27:
/Users/microbit-carlos/workspace/mbef/codal/microbit-v2-samples/libraries/codal-microbit-v2/inc/bluetooth/MicroBitBLEManager.h:34:2: error: #error "Please define SOFTDEVICE_PRESENT in your configuration"
   34 | #error "Please define SOFTDEVICE_PRESENT in your configuration"
      |  ^~~~~
[ 79%] Building CXX object libraries/codal-microbit-v2/CMakeFiles/codal-microbit-v2.dir/source/MicroBitFlash.cpp.obj
[ 80%] Building CXX object libraries/codal-microbit-v2/CMakeFiles/codal-microbit-v2.dir/source/MicroBitLog.cpp.obj
make[2]: *** [libraries/codal-microbit-v2/CMakeFiles/codal-microbit-v2.dir/source/MicroBitAudio.cpp.obj] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 80%] Building CXX object libraries/codal-microbit-v2/CMakeFiles/codal-microbit-v2.dir/source/MicroBitMemoryMap.cpp.obj
make[1]: *** [libraries/codal-microbit-v2/CMakeFiles/codal-microbit-v2.dir/all] Error 2
make: *** [all] Error 2

[microbit-carlos]

To replicate the issue building with pxt locally and then confirming that updating CMake inside the docker container resolves it:

1. Clone the pxt repo and modify the pxtarget.json file as shown in this PR: https://github.com/microsoft/pxt-microbit/pull/4829/files

   git clone https://github.com/microsoft/pxt-microbit.git
   cd pxt-microbit
   # Now modify pxtarget.json

2. Build pxt for CODAL only, this will show the same error:

   sudo npm install -g pxt
   npm install
   cd libs/blocksprj
   rm -rf built
   PXT_FORCE_LOCAL=1 PXT_COMPILE_SWITCHES=csv---mbcodal pxt build

3. Manually run the docker command to confirm that the same error is replicable

   cd built/dockercodal
   docker run --rm -v $(pwd):/src:delegated -w /src -u build pext/yotta:latest python build.py

4. Update CMake inside the container before building (via python3 pip, which requires quite a few commands but works) and the build will succeed:

   docker run --rm -v $(pwd):/src:delegated -w /src --user="root" pext/yotta:latest bash -c "\
   apt update && \
   apt install -y python3-pip && \
   python3 -m pip install pip==10.0.1 && \
   python3 -m pip install --upgrade pip && \
   python3 -m pip install cmake && \
   python build.py"

[microbit-carlos]

Correction, I didn't read the CMake docs correctly 😓 and the 3.12 addition was file(TOUCH, not file(APPEND which should work in 3.8.

However, it's clear the issue is resolved updating CMake versions, so I've narrowed it down to the issue being present in v3.16.8 and fixed in the next release v3.17.0.

CMake 3.17.0 release notes: https://cmake.org/cmake/help/latest/release/3.17.html

So I'm guessing the issue is how __merged.config is generated:

The foreach() command learned a new ZIP_LISTS option to iterate over multiple lists simultaneously.

foreach( _key _val IN ZIP_LISTS FINAL_FIELDS FINAL_VALUES )
    set( "__merged.config.${_key}" "${_val}" )
endforeach()

[microbit-carlos]

Fixed in https://github.com/lancaster-university/codal-microbit-v2/commit/a61573a8ca62ecd79c7f4d032fcfb1e649193bbf 🎉