Build errors when trying to build on Arch Linux

[M-Reimer]

If I just run the make without passing any "-jx", then it seems like the compiling does work.

But with "make ... -j4" I get:

Removing libelf-0.8.13/po/Makefile.in
Removing libelf-0.8.13/po/de.msg
Removing libelf-0.8.13/po/de.po
Removing libelf-0.8.13/po/gmo2msg.c
Removing libelf-0.8.13/po/libelf.pot
Removing libelf-0.8.13/po/stamp-po
Removing libelf-0.8.13/stamp-h.in
STAGE: .stage.prepgit
STAGE: .stage.checkout
STAGE: .stage.patch
STAGE: .stage.LINUX.start
STAGE: .stage.LINUX.binutils-config
STAGE: .stage.LINUX.mkspiffs
STAGE: .stage.LINUX.esptool
STAGE: .stage.LINUX.mklittlefs
make: *** [Makefile:548: .stage.LINUX.esptool] Error 2
make: *** Waiting for unfinished jobs....
make: *** [Makefile:512: .stage.LINUX.mkspiffs] Error 2
make: *** [Makefile:530: .stage.LINUX.mklittlefs] Error 2

What's going wrong here and where do I find the needed info to see what went wrong.

IMHO your Makefile is way too silent. I really would prefer to have the info right on my shell and not spread over several log files...

[M-Reimer]

Seems like you miss some dependencies in your Makefile. If I run with "-j1", then I do get

STAGE: .stage.LINUX.binutils-make

which is missing with "-j4". Seems like some kind of race condition here.

Edit: Had to add the "-j4" after all "$(MAKE)" in your Makefile to finally get my CPU to max out without breaking everything.

[earlephilhower]

Very odd. I run with -j32 on my ancient 16-core desktop all the time (for both the Docker multi-platform run and the simple Linux-only build) and haven't ever seen that kind of issue. Your system probably hit a race condition I've not seen. :(

If you have a suggested missing dependency, please do let me know or send a PR!

The reason for the separate log files is that I really do have up to 32 CCs running in parallel, for 6 different architectures and 4-5 targets each, so w/o it debugging a problem is basically impossible due to everything mixing together. You're welcome to adjust the output macros in your own fork, of course.

[M-Reimer]

This is the build command I try with:

$ make GHUSER=earlephilhower REL=1.1.1 SUBREL=1 GHTOKEN=none GCC=10.2

And to be honest: I have no clue at all on how to debug a Makefile.

Adding the "-j4" to the $(MAKE) commands caused an error, too, so I run a build on only one CPU, now.

Maybe "gnu make" with all its parallel handling and stuff is the wrong tool for this kind of batch processing...

[earlephilhower]

Maybe "gnu make" with all its parallel handling and stuff is the wrong tool for this kind of batch processing...

I think the GNU guys would beg to differ. :) It really works just fine for me and lets me run the multi-arch builds in parallel w/o issue. And -j4 does iterate thru the binutils.make stage for me (Ubuntu 18.04, mostly up-to-date).

earle@server:~/src/esp-quick-toolchain$ make -j4
STAGE: .stage.download
STAGE: .clean.binutils-gdb.git
STAGE: .clean.gcc.git
STAGE: .clean.newlib.git
STAGE: .clean.lx106-hal.git
HEAD is now at e4bcc63 Merge pull request #11 from olaf-mandel/cleanup2
Removing log.stage.patch
STAGE: .clean.mkspiffs.git
HEAD is now at 7fefeac add license file
STAGE: .clean.esptool.git
HEAD is now at f80ae31 Update for cross-compiled builds
STAGE: .clean.mklittlefs.git
HEAD is now at 6b5c62d Merge pull request #13 from earlephilhower/notests
HEAD is now at 50bc8381c Fix #define isXXX_l method access to ctype[]
HEAD is now at 2ae0bdddef xtensa: ld: convert tests to run_dump_test
HEAD is now at d230f4322ce xtensa: report stack usage
STAGE: .stage.prepgit
STAGE: .stage.checkout
STAGE: .stage.patch
STAGE: .stage.LINUX.start
STAGE: .stage.LINUX.binutils-config
STAGE: .stage.LINUX.mkspiffs
STAGE: .stage.LINUX.esptool
STAGE: .stage.LINUX.mklittlefs
STAGE: .stage.LINUX.binutils-make

The dependencies work off of the .stage.XXXX 0-byte files. You might want to check that they are properly cleared on your system since the .stage.LINUX.binutils-make depends on .stage.LINUX.binutils-config (i.e. it waits for the config target to touch .stage.LINUX.binutils-config.

Also, this makefile is only going to work under a recent Linux variant. I expect it won't run under cygwin/etc. Other folks have been using it, though, without incident so I have some confidence there's nothing specific to my setup to get it to run.

Looking at your failure, it looks like esptool and other simple tools are not building. I suggest checking the logfile to see what's up there, as it might lead you to a solution. These tools are super-simple to build and shouldn't ever fail to compile.

[M-Reimer]

I get the same error with "make -j1".

  CXX    xml-support.o
  CXX    xml-syscall.o
  CXX    xml-tdesc.o
  CXX    xtensa-config.o
  CXX    xtensa-tdep.o
  GEN    init.c
  CXX    init.o
  CXXLD  gdb
/usr/bin/ld: gdb-dlfcn.o: in function `gdb_dlopen(char const*)':
/home/manuel/git/esp-quick-toolchain/repo/binutils-gdb-gnu/gdb/gdb-dlfcn.c:65: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: cannot find -lsource-highlight
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:1889: gdb] Error 1
make[3]: Leaving directory '/home/manuel/git/esp-quick-toolchain/arena.x86_64/binutils-gdb-gnu/gdb'
make[2]: *** [Makefile:10420: all-gdb] Error 2
make[2]: Leaving directory '/home/manuel/git/esp-quick-toolchain/arena.x86_64/binutils-gdb-gnu'
make[1]: *** [Makefile:849: all] Error 2
make[1]: Leaving directory '/home/manuel/git/esp-quick-toolchain/arena.x86_64/binutils-gdb-gnu'

[earlephilhower]

Those errors mean your system can't build an executable file, it's not makefile related.

I would debug that first. You can go into the repo/mklittlefs directory, for example, and just try building there to simplify debug.

[M-Reimer]

The error happened in binutils-gdb-gnu.

If I go to arena.x86_64/binutils-gdb-gnu and run "make" there, then it finishes building. But probably with different settings than you would use for building.

Any chance that you can set up an Arch Linux VM and try building from there? Seems like my tools are just too recent...

[M-Reimer]

I have this in the "mklittlefs" build log when using "-j4".

make[1]: warning: -j1 forced in submake: resetting jobserver mode.
make[1]: Entering directory '/home/manuel/git/esp-quick-toolchain/arena.x86_64/mklittlefs'
x86_64-linux-gnu-g++ -std=gnu++11 -Os -Wall   -Itclap -Iinclude -Ilittlefs -I. -D VERSION=\"0.2.3-33-g6b5c62d\" -D LITTLEFS_VERSION=\"v2.0.5\" -D BUILD_CONFIG=\"\" -D BUILD_CONFIG_NAME=\"-arduino-esp8266\" -D __NO_INLINE__ -D LFS_NAME_MAX=32   -c -o main.o main.cpp
make[1]: x86_64-linux-gnu-g++: No such file or directory
make[1]: *** [<builtin>: main.o] Error 127
make[1]: Leaving directory '/home/manuel/git/esp-quick-toolchain/arena.x86_64/mklittlefs'

So I guess you are right that the parallel processing is not the problem. Seems like your Makefile is somewhat limited to some kind of distribution and does not run on my Arch setup.

[earlephilhower]

Sorry, I don't use Arch Linux and don't have the spare cycles to debug that specific distro. Except for snaps and systemd, Ubuntu's really been good to me since 410. :)

You may be able to override the expected G++/GCC compiler names on the make command line (CPP=/path/to/g++, etc.).

I suggest you look at the Docker file mentioned in the README.md. This will give you an Ubuntu distro filesystem with expected paths and all the myriad of dependencies required for cross-compiling. Other than the ~2GB download, it's painless if you can run docker on your distro.

[M-Reimer]

This is the file list of the GCC package that I use:

https://www.archlinux.org/packages/core/x86_64/gcc/files/

There is no "x86_64-linux-gnu-g++" on Arch. What I have is "x86_64-pc-linux-gnu-g++". You have this as "ahost" in your Makefile, but I don't see any place where you check for that or switch from "host" to "ahost".

[M-Reimer]

And if this is fixed, then it fails with

make[1]: x86_64-pc-linux-gnu-strip: No such file or directory

I don't have a "strip" with any prefix at all.

Can you just omit the prefixes if a build is ran for the current system only?

[earlephilhower]

Sure, feel free to hack the settings in the Makefile to match your system. The top section defines a bunch of variables depending on the build target so it should be pretty obvious. Building GCC is not for the faint hearted. Built a cross-compiling GCC is about 2x as intricate.

That said, I'm not sure how you would end up w/o any strip at all. It's not technically necessary (just removes debug info and shrinks files) but I've never seen a UNIX-like w/o it.

[M-Reimer]

I have "strip" but it's just named "strip" and nothing else. There is none with any "x86_64..." prefix:

https://www.archlinux.org/packages/core/x86_64/binutils/files/

Any chance to just call plain old "strip" if no cross compilation is requested?

[earlephilhower]

Sure. If it doesn't grok the exe format, it will let you know and not harm it anyway.

[M-Reimer]

I meant if this can be added directly to the Makefile somehow without reworking too much. If a build is only requested for the current machine, then just calling "gcc", "g++", "strip", ... will always be right and points to the correct toolchain for the current architecture.

Found the reason for a second error. There is a dependency to "source-highlight". Has to be installed to compile gdb.

Edit: Have to correct that: Arch does not ship libraries for statically linking. Had to override LINUX_BFLAGS to be empty to link dynamically.

[M-Reimer]

Finished building now. Created me the Arduino toolchain in the "arduino" subdirectory, but I won't try this out today (getting late now).

Raw steps to build on Arch:

Create "bin" directory inside the source tree with the following contents:

$ ls -l bin
total 0
lrwxrwxrwx 1 manuel users 12  3. Dez 20:30 x86_64-linux-gnu-g++ -> /usr/bin/g++
lrwxrwxrwx 1 manuel users 12  3. Dez 20:30 x86_64-linux-gnu-gcc -> /usr/bin/gcc
lrwxrwxrwx 1 manuel users 11  3. Dez 20:42 x86_64-linux-gnu-ld -> /usr/bin/ld
lrwxrwxrwx 1 manuel users 14  3. Dez 20:31 x86_64-linux-gnu-strip -> /usr/bin/strip

Then run the following to activate this new bin-directory for our build:

export PATH=$PATH:$PWD/bin

Then use the following command for building:

$ make GHUSER=earlephilhower REL=1.1.1 SUBREL=1 GHTOKEN=none GCC=10.2 LINUX_BFLGS=" "

To be able to "make install" this, the Python script "patch_json.py" needs the python interpreter name to be changed from just python to python2 as "python" points to a Python 3.x version on Arch which will not execute this script.

"make install" will fail if another lx106 compiler is installed on the system (will use this and then fails to find libraries) so I uninstalled my lx106 compiler. Won't need it anyway as I want to compile with Arduino only.

[earlephilhower]

Awesome. Would you like to make a README.arch.md file in a PR? The python2 dependency should be fixed, thanks for noting it. We scrubbed the Arduino core late last year, but I didn't look here.

[M-Reimer]

Something is still wrong... For some reason some tools are missing in the final "arduino" directory.

For example "mkspiffs" is built and available in the source tree but it does not land in the "tar.gz" and so does not land in the "tools" directory in the "arduino" subdirectory...

Edit: Actually there is a separate tar.gz for mkspiffs and other tools but they are never extracted with "make install". Sure that nothing is missing there?

[earlephilhower]

Nothing is missing or wrong, just a sequence issue.

You need to make upload a release, publish it on GH, and then run make install since it actually tests the get.py script which will d/l from the GH repo you specify. OTW you can manually extract the tarballs in the tools subdir, but you will also then need to manually build the bearssl.a and lqip.a files.

[M-Reimer]

Where does "make install" test the get.py? Didn't find this part.

Wouldn't it be a useful addition to have a Arduino toolchain built for own local use without any uploading in between?

[earlephilhower]

Where does "make install" test the get.py? Didn't find this part.

Ah, my memory failed me. make install creates the updated JSON file which would then be git commit -a; git push to enable a PR (which would then do the get.py in CI).

Wouldn't it be a useful addition to have a Arduino toolchain built for own local use without any uploading in between?

Honestly, for me it wouldn't. The only reason to have a make install is to generate a legit PR to run in CI. To just test a new compiler, all you do is untar the tgz file in an existing repo and give it a try. In practice it's 30 mins start to finish, including upload time for a full-blown PR and about 10 for a Linux X84-64-only build, and many times I'm just trying out a newlib change which takes 30 seconds to make the .a files I need to patch in.

On your own repo, no reason you can't adjust it to do what you'd like, of course. But to just try things I'm not sure it's worth any grief over cd tools; rm -rf xtensa-*; tar xvf ... in the correct directory.

[M-Reimer]

Honestly, for me it wouldn't. The only reason to have a make install is to generate a legit PR to run in CI.

But it would help others, who want to use your build scripts. I guess only a minority will set up all that stuff including CI and stuff just to compile a toolchain. It would be pretty cool to just have a make target with throws together a toolchain which just can be used with Arduino IDE right away.

As I have no clue how your CI works I'll have to find out all the missing steps on my own, now.

In the "bearssl" directory several ".o" files are left and packaged later. Can those be removed? Why do you build bearssl at all? For me it seems like the "official" Arduino toolchain comes with just the sourcecode of bearssl.

[earlephilhower]

There is literally nothing other than tar xvf <tarball name> required to install the toolchain in your own Arduino directory. Please, if that's all you want to do just make and you're done, no make install needed.

As I also ported BearSSL and wrote the SSL classes on the Arduino, please also believe me that you need to build bearssl.a and put it in the libs directory. You can do that with a make install in the Arduino dir (after installing the toolchain). Same with LWIP, done my d-a-v. A make after installing the toolchain and you're good.

[M-Reimer]

There is literally nothing other than tar xvf <tarball name> required to install the toolchain in your own Arduino directory. Please, if that's all you want to do just make and you're done, no make install needed.

Didn't know that it would be that easy until I tried it that way and finally had a working ESP8266 toolchain after around 5 or so days of not knowing about your build helpers and constantly trying to build lx106 compilers with different patch sets and constantly failing to build newlib with dozens of instructions just to find out that my compilers don't work with Arduino.

So thank you for sharing your toolchain building scripts even though they are clearly meant to be used for your own workflow. With only a few minimal additional steps this allows to build own toolchains which is amazing...

As I also ported BearSSL and wrote the SSL classes on the Arduino, please also believe me that you need to build bearssl.a and put it in the libs directory.

Didn't want to question that. I just didn't see where anything, compiled from BearSSL, landed in the final toolchain. But thanks for clarifying that.

I ended up with using "make install" to build the missing libraries as this prepares the environment to make it build with the right compiler.

Why do you extract one of your tarballs into the "tools" directory right when doing "make install" but not any of the other three? Just out of curiosity. I'm sure there are good reasons for that.

Would you like to make a README.arch.md file in a PR?

I'm working on something more useful for Arch Linux users. Arch Linux has a build concept based on so-called "PKGBUILD files" which are special shellscripts which perform all the build steps automatically to finally create an installable system package. I'm close to finishing one for an ESP8266 toolchain.