Firmware compilation fails under Docker (Windows) with missing GLIBC_2.27

[beels]

If I try to build main (at e134db193e57d7ca8914869eb590ee202d751610) under docker (Docker Desktop on Windows), I get the following errors:

root@97bf8fa8bb78:/target# make clean; rm -rf build
root@97bf8fa8bb78:/target# make
lib/l_bootstrap.c:8:10: fatal error: build/crowlib.h: No such file or directory
 #include "build/crowlib.h"
          ^~~~~~~~~~~~~~~~~
compilation terminated.
lib/ii.c:9:10: fatal error: ../build/ii_c_layer.h: No such file or directory
 #include "../build/ii_c_layer.h" // GENERATED BY BUILD PROCESS
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
lib/lualink.c:44:10: fatal error: build/First.h: No such file or directory
 #include "build/First.h"
          ^~~~~~~~~~~~~~~
compilation terminated.
build_dir build/
l2h lua/asl.lua -> build/asl.h
util/luacc: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by util/luacc)
Makefile:318: recipe for target 'lua/asl.lua.h' failed
make: *** [lua/asl.lua.h] Error 1

[beels]

Note that if I check out v3.0.1, I get similar error messages about missing files in build/, but the compilation succeeds and the resulting binary works fine when installed on crow.

[ghost]

I've been trying to get this working too, to add a lil' custom teensyLC module to the ii devices :) and I think the issue is the Docker file?

Using this as Dockerfile in the main directory seems to get v4.0.1 to compile:

FROM ubuntu:bionic

RUN apt-get update && \
    apt-get install -y --no-install-recommends software-properties-common && \
    add-apt-repository ppa:team-gcc-arm-embedded/ppa -y && \
    apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    bzip2 \
    dfu-util \
    gcc-arm-embedded \
    git \
    libreadline-dev \
    unzip \
    zip \
    xxd \
    wget && \
    rm -rf /var/lib/apt/lists/*

RUN wget --quiet https://www.lua.org/ftp/lua-5.3.4.tar.gz -O lua.tar.gz
RUN tar -xzf lua.tar.gz && \
    cd lua-5.3.4 && \
    make linux test && \
    make install && \
    cd ..

WORKDIR /target
ENTRYPOINT ["make", "-j", "R=1", "zip"]

It just updates the ubuntu release (fixing GLIBC_2.27 error) and adds the xxd package. Things then seem to move a bit further in compiling but v4.0.3 and main still seem to fail. The earlier No such file or directory error messages are still there at the start of the output, but then there's this final error at the end of the output:

lib/l_bootstrap.c:53:54: error: initializer element is not constant
     , { "lua_hotswap"   , build_hotswap_lc   , true, build_hotswap_lc_len}
                                                      ^~~~~~~~~~~~~~~~~~~~
lib/l_bootstrap.c:53:54: note: (near initialization for 'Lua_libs[13].len')
Makefile:274: recipe for target 'lib/l_bootstrap.o' failed
make: *** [lib/l_bootstrap.o] Error 1

[eethann]

I'm having the same experience: the updated Dockerfile allows compilation to start, but I error out with the same messages as above. Here are a few more lines from my terminal output for context (there are around 10 or so of these errors in the log before it exits):

submodules/STM32CubeF7_Drivers/STM32F7xx_HAL_Driver/Src/stm32f7xx_ll_usb.o
lib/detect.o
lib/repl.c: In function 'REPL_script_name_from_mem':
lib/repl.c:216:32: warning: unknown option after '#pragma GCC diagnostic' kind [-Wpragmas]
 #pragma GCC diagnostic ignored "-Wstringop-truncation"
                                ^~~~~~~~~~~~~~~~~~~~~~~
lib/repl.o
lib/ftrack.o
lib/l_bootstrap.c:39:54: error: initializer element is not constant
     { { "lua_crowlib"   , build_crowlib_lc   , true, build_crowlib_lc_len}
                                                      ^~~~~~~~~~~~~~~~~~~~
lib/l_bootstrap.c:39:54: note: (near initialization for 'Lua_libs[0].len')
lib/l_bootstrap.c:40:54: error: initializer element is not constant
     , { "lua_asl"       , build_asl_lc       , true, build_asl_lc_len}
                                                      ^~~~~~~~~~~~~~~~
lib/l_bootstrap.c:40:54: note: (near initialization for 'Lua_libs[1].len')
lib/l_bootstrap.c:41:54: error: initializer element is not constant
     , { "lua_asllib"    , build_asllib_lc    , true, build_asllib_lc_len}

[eethann]

Interestingly, I can get it to compile in an Ubuntu container in WSL2 just fine. The issue may be with the docker container? Or perhaps the mount directory? Compiling the first time I did get a bunch of errors about missing .h files corresponding to lua files, but they were then auto generated and compilation proceeded.

[trentgill]

quick notes to explain what is (and isn't) an issue here:

the following errors are expected the first time you compile. they're caused by issues in the Makefile (dependencies with the code generation isn't clear i think), but they don't stop the compilation, so i haven't prioritized fixing them:

lib/l_bootstrap.c:8:10: fatal error: build/crowlib.h: No such file or directory
 #include "build/crowlib.h"
          ^~~~~~~~~~~~~~~~~
compilation terminated.
lib/ii.c:9:10: fatal error: ../build/ii_c_layer.h: No such file or directory
 #include "../build/ii_c_layer.h" // GENERATED BY BUILD PROCESS
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
lib/lualink.c:44:10: fatal error: build/First.h: No such file or directory
 #include "build/First.h"
          ^~~~~~~~~~~~~~~
compilation terminated.

@beels

util/luacc: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by util/luacc)
Makefile:318: recipe for target 'lua/asl.lua.h' failed
make: *** [lua/asl.lua.h] Error 1

i had to write a lua bytecode cross-compiler (luacc) which is just a binary i included in the repo. it was built on an ubuntu x86_64 machine, so i assume there's issues bc of the internal architecture of the docker container.

the best(?) solution would be to add compilation of luacc (https://github.com/trentgill/lua/tree/luacc-5.3.4) into the crow build process, so the cross-compiler is native to the environment. i'm not sure if this is a drop-in solution, and i will try and find some notes on how to even compile it at all!

as to "why" this is necessary -- with all the new v4 features, we were running out of RAM constantly. previously we would save lua scripts as raw strings, then compile them while crow was running. i changed this so that we compile to lua bytecode on the PC, and save that bytecode into flash directly. this dramatically reduces RAM usage at runtime because we don't have to load the big strings into RAM and compile them.

the issue is crow runs on a 32bit ARM system, but we're (typically) compiling on a 64bit x86 machine. as such we need a cross-compiler to get the lua bytecode into the correct format so it runs on crow.

if there's another solution for this cross-compilation that doesn't require a compiled C program to do the hard work (it's just a modified version of luac from the lua repo) that'd be ideal. i spent a few days on it and couldn't find anything better.

@eethann #pragma GCC diagnostic ignored "-Wstringop-truncation" this can be ignored. ironically the #pragma is there to avoid a compiler warning, but is creating one for you :/

lib/l_bootstrap.c:39:54: error: initializer element is not constant
     { { "lua_crowlib"   , build_crowlib_lc   , true, build_crowlib_lc_len}
                                                      ^~~~~~~~~~~~~~~~~~~~
lib/l_bootstrap.c:39:54: note: (near initialization for 'Lua_libs[0].len')
lib/l_bootstrap.c:40:54: error: initializer element is not constant
     , { "lua_asl"       , build_asl_lc       , true, build_asl_lc_len}
                                                      ^~~~~~~~~~~~~~~~
lib/l_bootstrap.c:40:54: note: (near initialization for 'Lua_libs[1].len')
lib/l_bootstrap.c:41:54: error: initializer element is not constant
     , { "lua_asllib"    , build_asllib_lc    , true, build_asllib_lc_len}

this looks like the code-generation hasn't run properly before arriving here? build_crowlib_lc (for eg) is a lua bytecode file that is generated by luacc, so if it's "not constant" i assume that means the bytecode hasn't been generated.

---

hope it gives some perspective, and if anyone has ideas or finds a good solution i'd love to have it incorporated into the repo. added benefit is this would mean the github CI checks would start working again!