Building QT5 fails on Debian/stretch

brianmay commented 6 years ago

As below:

$ mix nerves.artifact
|Elixir.Docker| Starting Build... (this may take a while)

>>> qt5base 5.11.1 Configuring** (Mix) The Nerves Docker build_runner encountered an error while building:

-----
  GEN     /nerves/build/Makefile
>>> qt5base 5.11.1 Configuring
mkdir -p /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/
sed 's/@EGLFS_DEVICE@//g' package/qt5/qt5base//qmake.conf.in > /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/qmake.conf
/usr/bin/install -m 0644 -D package/qt5/qt5base//qplatformdefs.h /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/qplatformdefs.h
touch /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/arch.conf
printf 'LIBS += -latomic\n' >/nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/arch.conf
(cd /nerves/build/build/qt5base-5.11.1; PATH="/nerves/build/host/bin:/nerves/build/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" PKG_CONFIG="/nerves/build/host/bin/pkg-config" PKG_CONFIG_LIBDIR="/nerves/build/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/pkgconfig" PKG_CONFIG_SYSROOT_DIR="/nerves/build/host/arm-buildroot-linux-gnueabihf/sysroot" MAKEFLAGS=" --no-print-directory --  -j8" ./configure -v -prefix /usr -hostprefix /nerves/build/host -headerdir /usr/include/qt5 -sysroot /nerves/build/host/arm-buildroot-linux-gnueabihf/sysroot -plugindir /usr/lib/qt/plugins -examplesdir /usr/lib/qt/examples -no-rpath -nomake tests -device buildroot -device-option CROSS_COMPILE="/nerves/build/host/bin/arm-unknown-linux-gnueabihf-" -device-option BR_COMPILER_CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os " -device-option BR_COMPILER_CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os " -optimized-qmake -no-cups -no-iconv -system-zlib -system-pcre -no-pch -shared  -no-kms -release -opensource -confirm-license -no-sql-mysql -no-sql-psql  -system-sqlite  -gui -system-freetype  -no-harfbuzz -widgets --enable-linuxfb -no-directfb -no-xcb -opengl es2 -qpa eglfs -eglfs -openssl -fontconfig  -no-libjpeg -system-libpng -dbus -no-tslib -glib -icu -nomake examples -no-libinput -no-gtk )
Creating qmake...
make[2]: Nothing to be done for 'first'.
Could not find qmake spec ''.
Error processing project file: /nerves/build/build/qt5base-5.11.1
package/pkg-generic.mk:213: recipe for target '/nerves/build/build/qt5base-5.11.1/.stamp_configured' failed
make[1]: *** [/nerves/build/build/qt5base-5.11.1/.stamp_configured] Error 3
make: *** [_all] Error 2
Makefile:16: recipe for target '_all' failed

-----

See /home/brian/tree/personal/relixir/kiosk_system_rpi3/build.log.

dylanfareed commented 6 years ago

I'm seeing the same on macOS High Sierra (and have been unable to build kiosk_system_rpi3 locally for some time. These are the contents of the build.log on my machine.

Setting User
UID: 501
GID: 20
Switching user
make: Entering directory '/nerves/env/platform/buildroot-2018.05.1'
  GEN     /nerves/build/Makefile
#
# configuration written to /nerves/build/.config
#
make: Leaving directory '/nerves/env/platform/buildroot-2018.05.1'
------------

Build directory successfully created.

Configuration: /nerves/env/kiosk_system_rpi3/nerves_defconfig

Next, do the following:
   1. cd /nerves/build
   2. make

For additional options, run 'make help' in the build directory.

IMPORTANT: If you update nerves_system_br, you should rerun this script.
Setting User
UID: 501
GID: 20
Switching user
umask 0022 && make -C /nerves/env/platform/buildroot-2018.05.1 O=/nerves/build/. 
/usr/bin/make -j1 O=/nerves/build HOSTCC="/usr/bin/gcc" HOSTCXX="/usr/bin/g++" silentoldconfig
  GEN     /nerves/build/Makefile
>>> qt5base 5.11.1 Configuring
mkdir -p /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/
sed 's/@EGLFS_DEVICE@//g' package/qt5/qt5base//qmake.conf.in > /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/qmake.conf
/usr/bin/install -m 0644 -D package/qt5/qt5base//qplatformdefs.h /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/qplatformdefs.h
touch /nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/arch.conf
printf 'LIBS += -latomic\n' >/nerves/build/build/qt5base-5.11.1/mkspecs/devices/linux-buildroot-g++/arch.conf
(cd /nerves/build/build/qt5base-5.11.1; PATH="/nerves/build/host/bin:/nerves/build/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" PKG_CONFIG="/nerves/build/host/bin/pkg-config" PKG_CONFIG_LIBDIR="/nerves/build/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/pkgconfig" PKG_CONFIG_SYSROOT_DIR="/nerves/build/host/arm-buildroot-linux-gnueabihf/sysroot" MAKEFLAGS=" --no-print-directory --  -j8" ./configure -v -prefix /usr -hostprefix /nerves/build/host -headerdir /usr/include/qt5 -sysroot /nerves/build/host/arm-buildroot-linux-gnueabihf/sysroot -plugindir /usr/lib/qt/plugins -examplesdir /usr/lib/qt/examples -no-rpath -nomake tests -device buildroot -device-option CROSS_COMPILE="/nerves/build/host/bin/arm-unknown-linux-gnueabihf-" -device-option BR_COMPILER_CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os " -device-option BR_COMPILER_CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os " -optimized-qmake -no-cups -no-iconv -system-zlib -system-pcre -no-pch -shared  -no-kms -release -opensource -confirm-license -no-sql-mysql -no-sql-psql  -system-sqlite  -gui -system-freetype  -no-harfbuzz -widgets --enable-linuxfb -no-directfb -no-xcb -opengl es2 -qpa eglfs -eglfs -openssl -fontconfig  -no-libjpeg -system-libpng -dbus -no-tslib -glib -icu -nomake examples -no-libinput -no-gtk )
Creating qmake...
make[2]: Nothing to be done for 'first'.
Could not find qmake spec ''.
Error processing project file: /nerves/build/build/qt5base-5.11.1
make[1]: *** [/nerves/build/build/qt5base-5.11.1/.stamp_configured] Error 3
package/pkg-generic.mk:213: recipe for target '/nerves/build/build/qt5base-5.11.1/.stamp_configured' failed
make: *** [_all] Error 2
Makefile:16: recipe for target '_all' failed

fhunleth commented 6 years ago

Is there a reason that the prebuilt kiosk_system_rpi3 doesn't work for you? It sounds like there's an issue with the Nerves Docker integration for Mac. That should be fixed. However, if you skip past that, you're looking at what might be a 12 hour build time for Qt5 and Chromium.

brianmay commented 6 years ago

In my case I need extra stuff that isn't provided by the default image. Such as espeak, mpd and mpc.

fhunleth commented 6 years ago

I think what may be happening is that QMAKESPEC is defined and it's messing up Buildroot when it goes to compile Qt. Oddly enough, I can't get that to fail in Linux.

dylanfareed commented 6 years ago

Thanks @fhunleth. I'm in the same boat as @brianmay; I'm trying to add wifi & sound drivers for a custom cm3 carrier board.

The errors here tho were from building against master and unchanged aside from commenting out the GitHub artifacts.

Thanks so much regardless!

GregMefford commented 6 years ago

@dylanfareed As a simple troubleshooting step, can you try docker pull nerves_system_br:latest to make sure you have the latest version of the container that runs the Nerves system shell?

GregMefford commented 6 years ago

@brianmay Are you running this on OSX, or on Debian (as suggested by the issue title)? Unless you've overridden something, I'd be surprised that Docker is being used on Debian.

brianmay commented 6 years ago

Yes, this is Debian. Not sure if the build is using Docker or not. I do see I have version 1.4.5 of the image installed, so it could actually be using it.

dylanfareed commented 6 years ago

Thanks @GregMefford! I've run docker pull nervesproject/nerves_system_br:latest and got the same error unfortunately.

This was the state of my system previously

And now

The error was the same again however

...

I am going to bump the mix file dep for nerves_system_br up to 1.5.1 and try again (and if that doesn't work I'll try some older versions as this used to work alright).

FWIW I am running mix compile here. I am not sure if there's a different command I should be running in its stead.

Thanks!

GregMefford commented 6 years ago

I'm also curious whether something fishy is happening in the build environment due to something having been compiled previously and it's not recompiling properly. I believe if you run mix nerves.system.shell to get into the container environment, then make clean, it should reset everything to do a clean build.

From there, you can exit from the system shell and try mix nerves.artifact. I believe this will take something like 10-12 hours if successful. This command should do the same thing as mix compile except that it will dump out the system archive file at the end so you can easily distribute it as needed.

dylanfareed commented 6 years ago

Thanks @GregMefford. That's a good thought. I've given that a shot to no avail unfortunately. The compile still fails in the same way.

The entire current diff for my local repo (just commenting out artifacts)

The machine I am using to produce the build

Docker for Mac version (latest stable)

Docker for Mac memory config

Terminal output of mix nerves.system.shell and make clean (which I think is what I should expect to see).

And finally the output from mix nerves.artifact

...

... 1.5 hours later ...

GregMefford commented 6 years ago

I was able to replicate the problem on my end. I'm still not sure what the problem is, but at least I can get it to fail on my machine. 👍 😁

dylanfareed commented 6 years ago

Awesome. Thanks @GregMefford!

I'm going to try to build on AWS CodeBuild. Will report back on how that goes.

dylanfareed commented 6 years ago

I've set up a repo to enable testing building the project on AWS CodeBuild and I'm tracking progress in this issue https://github.com/dylanfareed/ks/issues/1

The first build attempt failed somewhat surprisingly (at the same step but with different error output) because filename being created was too long. https://github.com/dylanfareed/ks/issues/1#issuecomment-425159092

Renaming the project got us past the qt5base 5.11.1 Configuring step but the build is ongoing (and will likely just timeout when we get to the 8 hour codebuild limit).

Time will tell. Thanks!

dylanfareed commented 6 years ago

Well, the build on CodeBuild appears to have succeeded

https://github.com/dylanfareed/ks/issues/1#issuecomment-425221028

I may proceed in that direction rather than building artifacts locally for the time being as it also appears to be much quicker--3 hours 22 minutes.

Prior to that, on a lark, I'll try shortening the repo name and building that locally again.

Thanks for folks eyes on this at this point.

GregMefford commented 6 years ago

This looks like it's not an issue with Nerves per se, but an issue with building QT in a Docker container. Based on a few forum threads, my guess is that we need to update libseccomp in the nerves_system_br container in order for this to work, but the latest official package isn't new enough, so we'll have to build it from source or something. I'm timed out for now, but I'll try looking into that later.

I did already try adding --security-opt seccomp:unconfined to the docker run invocation in Nerves, based on one of the forum posts, but it didn't help.

Breadcrumbs if someone else wants to look into it:

dalpo commented 6 years ago

Hello everyone!

I encountered the same issues and hardconding the --security-opt seccomp:unconfined option to the docker run invocation worked for me.

This is the change to the nerves dependency:

File path: nerves/lib/nerves/artifact/build_runners/docker.ex

defp run(pkg, cmd, stream) do
    set_volume_permissions(pkg)

    {_dockerfile, image} = config(pkg)

    args =
      [
        "run",
        "--rm",
        "-w=#{@working_dir}",
        "-a",
        "stdout",
        "-a",
        "stderr",
        "--security-opt",      # Added line
        "seccomp:unconfined"   # Added line
      ] ++ env() ++ mounts(pkg) ++ ssh_agent() ++ [image | cmd]
    # ...
  end

Note: I'm using nerves_system_br v1.5.2.

I hope it could help someone as a temporary workaround.

GregMefford commented 5 years ago

Oh, interesting! I also tried that and it didn't work for me. I assume that something different about my environment made it not work properly, or something was cached/already running that made the change not take effect.

I think this isn't a great "official fix," but I'm glad it works as a temporary solution and the actual security impact for this use-case is negligible because it's only going to matter during the local Nerves system build process; not on an Internet-facing service or anything like that. 👍 🚀

GregMefford commented 5 years ago

Confirmed to be still a problem on Docker engine 18.09.0. I'm working on an update to the nervesproject/nerves_system_br container that might fix it, but I'm currently not clear on whether this is fixable from inside the container or if we have to wait for some other collection of stars to align.

Essentially, we're waiting for this code to appear in the Docker engine and (I think) also the code inside the container needs to have a compatible version of libseccomp (>= 2.3.3). This is only available in pretty recent versions of Ubuntu, thus the update to the container we use for Nerves.

robvandenbogaard commented 5 years ago

@GregMefford I am also struggling with this error (aiming to compile a customized image - but so far unchanged kiosk_system_rpi3 in the same manner as described by Dylan). Last thing I tried was pulling latest nerves_system_br (v1.6.6) and upgrading all dependencies. I'm on Ubuntu 18.04. I am now trying dalpo's workaround. EDIT1: The workaround enables me at least to successfully get past >>> qt5base 5.11.2 Configuring - fingers crossed for the rest of the build ;) EDIT2: I was able to build until invocation of qmake which seemed to be looking for qt4 (specifically /usr/lib/x86_64-linux-gnu/qt4/bin/qmake) - a sudo apt install qt5-default fixed this.

Summarizing: I am able to build kiosk_system_rpi3 locally now on Ubuntu 18.04, by

after installing docker (snap), creating a docker user group and adding myself to it, to get access to the docker socket
pulling latest nerves_system_br
upgrading all dependencies
setting qt5 qmake as default: sudo apt install qt5-default
adding the security option to unconfine docker for the build in nerves/lib/nerves/artifact/build_runners/docker.ex within the deps folder for the system in the project: --security-opt seccomp:unconfined as described above by dalpo
move some files away to have enough disk space... :P

nerves-web-kiosk / kiosk_system_rpi3

Building QT5 fails on Debian/stretch #37