search

JuliaPackaging / BinaryBuilder.jl

Binary Dependency Builder for Julia
https://binarybuilder.org
Other
392 stars 101 forks source link

LoadError: Your platform ("x86_64-pc-linux-gnu", parsed as "x86_64-apple-darwin14") is not supported by this package! #408

Closed kdheepak closed 5 years ago

kdheepak commented 5 years ago

I get the following error when I try to build my CPP package on Travis-CI.

 LoadError: LoadError: Your platform ("x86_64-pc-linux-gnu", parsed as "x86_64-apple-darwin14") is not supported by this package!

Any suggestions on how to debug this? I have the environment variable BINARYBUILDER_AUTOMATIC_APPLE=true set.

https://travis-ci.com/GMLC-TDC/HELICSBuilder/builds/102397117#L546

kdheepak commented 5 years ago

I'm having various issues when trying to compile BinaryBuilder.jl locally and on Travis. The above was when I was trying to build a C++17 application on Travis for MacOS. I still don't know how to resolve this above issue. But instead of creating new issues for the other problems I'm having, I'm going to comment here. If I identify something that needs fixing in BinaryBuilder, then I'll create a new issue. For now I'm just assuming that I'm doing something incorrectly.

I am now trying to compile the same application for Linux on Travis. I need to compile HELICS with c++17, and cannot use a gcc 4.8.x compiler for this.

This is the output when I use Linux(:x86_64) as the only platform.

https://travis-ci.com/GMLC-TDC/HELICSBuilder/builds/102527399

As you can see on Travis, it gets past the CMake step, but fails during the make step, because there's invalid syntax.

/workspace/srcdir/HELICS-src-2.0.0-rc1/src/helics/application_api/../core/federate_id.hpp:119:54: warning: missing terminating ' character [enabled by default]
     bool isValid () const { return (gid != -2'010'000'000); }

If I change the compiler abi to gcc7, I get the following error: https://travis-ci.com/GMLC-TDC/HELICSBuilder/builds/102528894#L1816

ERROR: LoadError: LoadError: Your platform x86_64-linux-gnu-gcc7 is not supported by this package!

I'm using Boost and ZMQ from the following locations:

dependencies = [
               "https://github.com/JuliaInterop/ZMQBuilder/releases/download/v4.2.5%2B6/build.jl",
               "https://github.com/benlorenz/boostBuilder/releases/download/v1.67.0/build_boost.v1.67.0.jl",
              ]

The following is the full script I'm using build_tarballs.jl

using BinaryBuilder

src_tarball = "https://github.com/gmlc-tdc/helics-src/archive/v2.0.0-rc1.tar.gz"
src_hash    = "47524ea2558961a6b924a1f972b2d063ca43daa092d91b3b8865ae06f5b33961"
sources = [
    src_tarball => src_hash,
]

script = raw"""
cd $WORKSPACE/srcdir
cd HELICS-src-2.0.0-rc1
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=/opt/$target/$target.toolchain ..
make
make install
"""

products(prefix) = [
    LibraryProduct(prefix, "libhelicsSharedLib", :libhelicsSharedLib),
]

platforms = [
             Linux(:x86_64, compiler_abi=CompilerABI(:gcc7))
]

dependencies = [
               "https://github.com/JuliaInterop/ZMQBuilder/releases/download/v4.2.5%2B6/build.jl",
               "https://github.com/benlorenz/boostBuilder/releases/download/v1.67.0/build_boost.v1.67.0.jl",
              ]

# Build 'em!
build_tarballs(
    ARGS,
    "libhelicsSharedLib",
    v"2.0.0-rc1",
    sources,
    script,
    platforms,
    products,
    dependencies,
)

Any suggestions on what this error means?

tshort commented 5 years ago

I don't know about the error. It can often help to have ARGS = ["--debug"] in the call to build_tarballs. Then when it errors, it drops you into a shell for the target that had the error. Then, you can manually try the build and look at various gcc issues.

You might also try the Mac target. Another option is to try building with clang. I don't know if that is an option or if any other builders just use clang.

kdheepak commented 5 years ago

Thanks for the comment @tshort. I think one of the issues is that the boost dependency isn't built for gcc6. I think that people should add the following, in order to make the binaries work across more versions: 

platforms = expand_gcc_versions(platforms)

I think I may have just got a Linux version of HELICS building on Travis. I'm working through the Mac version at the moment.

kdheepak commented 5 years ago

None of it works locally on my Mac though. When I run the following:

$ julia-1.1 build_tarballs.jl --verbose --debug
.....
[16:30:00] # Stop if we hit any errors.
[16:30:00] set -e
[16:30:00]
[16:30:00] # Swap out srcdir from underneath our feet
[16:30:00] tmpify_srcdir
[16:30:00]
[16:30:00] cd $WORKSPACE/srcdir
[16:30:00] cd HELICS-src-2.0.0-rc1
[16:30:00] mkdir build
[16:30:00] cd build
[16:30:00] cmake -DZMQ_USE_STATIC_LIBRARY=ON -DAUTOBUILD_ZMQ=ON -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=/opt/$target/$target.toolchain ..
[16:30:00] make
[16:30:00] make install
[16:30:00]
[16:30:00] `
[16:30:00]  ---> trap "save_env" EXIT
[16:30:00]  ---> trap "save_env; save_srcdir" INT TERM ERR
[16:30:00]  ---> set -e
[16:30:00]  ---> tmpify_srcdir
[16:30:00] Copying srcdir to tmpfs...
[16:30:00] rsync: pipe: Address family not supported by protocol (97)
[16:30:00] rsync error: error in IPC code (code 14) at pipe.c(122) [sender=3.1.3]
[16:30:00] reboot: Power down
┌ Warning: Build failed, launching debug shell
└ @ BinaryBuilder ~/.julia/packages/BinaryBuilder/vkHhH/src/AutoBuild.jl:561
Setting parent shell interrupt to ^] (use ^C as usual, ^] to interrupt Julia)
verbose sandbox enabled (running in init mode)
Parsed --cd as "/workspace/srcdir"
Parsed --workspace as "9p/workspace0" -> "/workspace"
Parsed --workspace as "9p/workspace1" -> "/meta"
Parsed --map as "/dev/vdb" -> "/opt/x86_64-linux-gnu/BaseCompilerShard-2018.11.11"
Parsed --map as "/dev/vdc" -> "/opt/x86_64-linux-gnu/GCC-6.1.0"
Parsed --map as "/dev/vdd" -> "/opt/x86_64-linux-gnu/LLVM-6.0.1-0"
Reading 80 environment mappings
--> Creating overlay workdir at /proc
--> Mounting overlay of / at /tmp (modifications in /proc/upper/rootfs, workspace in /proc/work/rootfs)
--> mapping /dev/vdd to /tmp/opt/x86_64-linux-gnu/LLVM-6.0.1-0
--> mapping /dev/vdc to /tmp/opt/x86_64-linux-gnu/GCC-6.1.0
--> mapping /dev/vdb to /tmp/opt/x86_64-linux-gnu/BaseCompilerShard-2018.11.11
--> Mounting overlay of /tmp/opt/x86_64-linux-gnu/BaseCompilerShard-2018.11.11:/tmp/opt/x86_64-linux-gnu/GCC-6.1.0:/tmp/opt/x86_64-linux-gnu/LLVM-6.0.1-0 at /tmp/opt/x86_64-linux-gnu (modifications in /proc/upper/x86_64-linux-gnu, workspace in /proc/work/x86_64-linux-gnu)
--> Mounting procfs at /tmp/proc
--> Mounting /dev at /tmp/dev
--> workspacing 9p/workspace1 to /tmp/meta
--> workspacing 9p/workspace0 to /tmp/workspace
--> Mounting procfs at /proc
About to run `/bin/bash` `--init-file` `/meta/.env`
sandbox:$WORKSPACE/srcdir# ls
HELICS-src-2.0.0-rc1
sandbox:$WORKSPACE/srcdir# cd HELICS-src-2.0.0-rc1/
sandbox:$WORKSPACE/srcdir/HELICS-src-2.0.0-rc1# ls
CHANGELOG.md     DISCLAIMER       appveyor.yml     interfaces
CMakeLists.txt   LICENSE          config           scripts
CONTRIBUTING.md  README.md        docs             src
CONTRIBUTORS.md  ThirdParty       examples         tests
sandbox:$WORKSPACE/srcdir/HELICS-src-2.0.0-rc1# mkdir build
sandbox:$WORKSPACE/srcdir/HELICS-src-2.0.0-rc1# cd build/
sandbox:$WORKSPACE/srcdir/HELICS-src-2.0.0-rc1/build# cmake --version
cmake version 3.11.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
Segmentation fault
=/opt/$target/$target.toolchain .._INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=
-- The C compiler identification is GNU 6.1.0
-- The CXX compiler identification is GNU 6.1.0
-- Check for working C compiler: /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc
-- Check for working C compiler: /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-g++
-- Check for working CXX compiler: /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- setting up for GNU
-- setting version build options to "-std=c++14"
CMake Warning at /usr/share/cmake/Modules/FindBoost.cmake:839 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindBoost.cmake:950 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
  config/cmake/addBoost.cmake:111 (find_package)
  CMakeLists.txt:282 (include)

CMake Warning at /usr/share/cmake/Modules/FindBoost.cmake:839 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindBoost.cmake:950 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
  config/cmake/addBoost.cmake:111 (find_package)
  CMakeLists.txt:282 (include)

CMake Warning at /usr/share/cmake/Modules/FindBoost.cmake:839 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindBoost.cmake:950 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
  config/cmake/addBoost.cmake:111 (find_package)
  CMakeLists.txt:282 (include)

CMake Warning at /usr/share/cmake/Modules/FindBoost.cmake:839 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindBoost.cmake:950 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
  config/cmake/addBoost.cmake:111 (find_package)
  CMakeLists.txt:282 (include)

-- Boost version: 1.69.0
-- Found the following Boost libraries:
--   program_options
--   filesystem
--   system
--   unit_test_framework
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- cxx compiler /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-g++
-- linker /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld
-- flags  -fPIC -std=c++14
-- Configuring ZeroMQ Autobuild for  logging to /workspace/srcdir/HELICS-src-2.0.0-rc1/build/logs/zmq_autobuild_config.log
-- Building ZeroMQ  build logging to /workspace/srcdir/HELICS-src-2.0.0-rc1/build/logs/zmq_autobuild_build.log
make: Warning: File 'Makefile' has modification time 1.4 s in the future
make: *** [Makefile:146: cmake_check_build_system] Segmentation fault
-- Could NOT find ZeroMQ (missing: ZeroMQ_DIR)
-- Looking for open
-- Looking for open - found
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/srcdir/HELICS-src-2.0.0-rc1/build
Segmentation fault
sandbox:$WORKSPACE/srcdir/HELICS-src-2.0.0-rc1/build#

cmake and make segfault every time. I'm not really sure what is going on here. cmake seems to be working though, it generates the Makefiles but running make on them still segfaults.

kdheepak commented 5 years ago

Boost builds on MacOS fail because it is not able to find cc. This is the output from bootstrap.log.

###
### Using 'darwin' toolset.
###
rm -rf bootstrap
mkdir bootstrap
cc -o bootstrap/jam0 command.c compile.c constants.c debug.c execcmd.c framfunction.c glob.c hash.c hdrmacro.c headers.c jam.c jambase.c jamgram.c lismake.c make1.c object.c option.c output.c parse.c pathsys.c regexp.c rules.n.c search.c subst.c timestamp.c variable.c modules.c strings.c filesys.c bns.c class.c cwd.c native.c md5.c w32_getreg.c modules/set.c modules/path.cles/regex.c modules/property-set.c modules/sequence.c modules/order.c execu fileunix.c pathunix.c
./build.sh: line 336: cc: not found
kdheepak commented 5 years ago

There appears to be a $CC defined.

sandbox:$WORKSPACE/srcdir/boost_1_69_0# echo $CC
/opt/x86_64-linux-gnu/tools/clang -target x86_64-apple-darwin14 --sysroot /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root
sandbox:$WORKSPACE/srcdir/boost_1_69_0#

It seems to be clang, so I tried to set the toolset with clang. But I still get an error. If I run ./bootstrap.sh --with-toolset=clang I get the following bootstrap.log:

###
### Using 'clang' toolset.
###
rm -rf bootstrap
mkdir bootstrap
clang -Wno-unused -Wno-format -o bootstrap/jam0 command.c compile.c constants.c
debug.c execcmd.c frames.c function.c glob.c hash.c hdrmacro.c headers.c jam.c jj
ambase.c jamgram.c lists.c make.c make1.c object.c option.c output.c parse.c patt
hsys.c regexp.c rules.c scan.c search.c subst.c timestamp.c variable.c modules.cc
 strings.c filesys.c builtins.c class.c cwd.c native.c md5.c w32_getreg.c modulee
s/set.c modules/path.c modules/regex.c modules/property-set.c modules/sequence.cc
 modules/order.c execunix.c fileunix.c pathunix.c
In file included from command.c:17:
./jam.h:333:10: fatal error: 'sys/types.h' file not found
#include <sys/types.h>
         ^~~~~~~~~~~~~
1 error generated.
In file included from compile.c:23:
./jam.h:333:10: fatal error: 'sys/types.h' file not found
#include <sys/types.h>
         ^~~~~~~~~~~~~
1 error g
kdheepak commented 5 years ago

The solution for sys/types.h file not found appears to be to run xcode-select --install. But this cannot be run, right? This is a Linux machine that has the XCode SDK on it. Any suggestions on what to do here?

kdheepak commented 5 years ago

I've tried everything but am unable to cross compile boost in the Julia environment. The following is output from when I set the platform to be Windows. There seem to be garbage values in the bootstrap.log, so I'm not sure what is going on here.

sandbox:$WORKSPACE/srcdir/boost_1_68_0# ./bootstrap.sh --prefix=$prefix
Building Boost.Build engine with toolset gcc...
Failed to build Boost.Build build engine
Consult 'bootstrap.log' for more details
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0# cat bootstrap.log
###
### Using 'gcc' toolset.
###
rm -rf bootstrap
mkdir bootstrap
gcc -DNT -o bootstrap/jam0 command.c compile.c constants.c debug.c execcmd.c frames.c function.c glob.c hash.c hdrmacro.c headers.c jam.c jambase.c jamgram.c lists.c make.c make1.c object.c option.c output.c parse.c pathsys.c regexp.c rules.c scan.c search.c subst.c timestamp.c variable.c modules.c strings.c filesys.c builtins.c class.c cwd.c native.c md5.c w32_getreg.c modules/set.c modules/path.c modules/regex.c modules/property-set.c modules/sequence.c modules/order.c execnt.c filent.c pathnt.c
./bootstrap/jam0 -f build.jam --toolset=gcc --toolset-root= clean
./bootstrap/jam0: line 1: MZ@: not found
./bootstrap/jam0: line 4: : not found
./bootstrap/jam0: line 4: Eu: not found
./bootstrap/jam0: line 5: HEH}u
=w: not found                  3=H
./bootstrap/jam0: line 6: syntax error: unexpected ")"
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#
sandbox:$WORKSPACE/srcdir/boost_1_68_0#

To cross compile boost, one needs to run the ./bootstrap.sh first and then change some files manually. I'm not able to get the first ./bootstrap.sh to work when I use Julia and BinaryBuilder. I've tried the same thing on my Ubuntu virtual machine without using Julia and I the bootstrap process works file. 

parallels@parallels-vm:~/gitrepos/boostBuilder/boost_1_68_0$ ./bootstrap.sh 
Building Boost.Build engine with toolset gcc... tools/build/src/engine/bin.linuxx86_64/b2
Detecting Python version... 2.7
Detecting Python root... /usr
Unicode/ICU support for Boost.Regex?... not found.
Backing up existing Boost.Build configuration in project-config.jam.1
Generating Boost.Build configuration in project-config.jam...

Bootstrapping is done. To build, run:

    ./b2

To adjust configuration, edit 'project-config.jam'.
Further information:

   - Command line help:
     ./b2 --help

   - Getting started guide: 
     http://www.boost.org/more/getting_started/unix-variants.html

   - Boost.Build documentation:
     http://www.boost.org/build/doc/html/index.html

parallels@parallels-vm:~/gitrepos/boostBuilder/boost_1_68_0$

Any suggestions here would be appreciated. I'm not really sure where to go from here.

kdheepak commented 5 years ago

I got all of this to work today. Changes in BinaryBuilder.jl + patches to force it to use gcc8, and changes in boostBuilder provided by @benlorenz seem to have resolved my issues on Linux and MacOS.