Error building Julia on Raspberry Pi zero (both Docker and locally) >= 1.2.0

[terasakisatoshi]

• I'm trying to Build Julia from source for Raspberry Pi zero. Since Raspberry Pi zero adopts armv6, the official binary for arm32 bit / armv7 does not run on it. This means I need build julia from source. So I've made repository to store Dockerfiles to build Julia for ARM devices including Raspberry Pi zero. I succeed build Julia its version is 1.0.x and 1.1.x (where x is minor version). See:
  • Julia-Embedded/jlcross
• On the other hand, I can't confirm building Julia >= 1.2.0 from source works fine because of getting message illegal instruction. See my gist with full output log.
• I do not think it is an issue of Docker/cross-compilation issue. In fact, I've already tried my building procedure on Raspberry Pi locally (i.e. without using Docker) and got similar error. Many error occurs near by

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/pi/work/build/usr/tools /home/pi/work/build/contrib/fixup-libgfortran.sh --verbose /home/pi/work/build/usr/lib
Discovered traces of libgfortran within . 
Got SONAMES of 

• which is a part of building log e.g.:

  • line 1774
  • building log locally

• I think there is a clue related to fixup-libgfortran.sh to solve our problem, but I have no idea how to solve it. Does anyone have a ray of hope to solve the issue ?

Thank you.

[ViralBShah]

Have you seen: https://github.com/JuliaLang/julia/blob/master/doc/build/arm.md ?

[terasakisatoshi]

I should have added link how to reproduce our issue

@ViralBShah Yes. This doc says:

Julia requires at least the armv6

So there is a chance to build Julia.

We discussed the issue several months ago. At this moment (1.1.x was the latest version of Julia )we could build Julia from source for all Raspberry Pi series.

[ViralBShah]

I wonder if disabling the use of binary builder helps? That way you will build llvm on the pi zero.

[terasakisatoshi]

I wonder if disabling the use of binary builder helps? That way you will build llvm on the pi zero.

This means add USE_BINARYBUILDER = 0 to Make.user via

$ echo "USE_BINARYBUILDER = 0" >> Make.user

?

I will try it.

[ViralBShah]

Yes, that's right. I am not sure how effective that will be, but it is worth a try.

[terasakisatoshi]

Sorry my delay of reply. Adding USE_BINARYBUILDER = 0 doesn't solve our problem 😭

Here is my dockerfile I used for reproducing error

# Build Julia binary for arm32-bit devices e.g. RaspberryPi Zero(W/WH)
FROM balenalib/raspberry-pi:buster-20191030

MAINTAINER SATOSHI TERASAKI

# install dependencies
RUN apt-get update && \
    apt-get install -y build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config \
    libopenblas-base libopenblas-dev \
    libatlas3-base libatlas-base-dev \
    liblapack-dev \
    libmpfr-dev libgmp3-dev \
    libgfortran3 \
    git

# build julia from source
ARG JL_VERSION="v1.2.0"
ARG WDIR=/home/pi/work
ARG JL_BUILD_DIR=$WDIR/build
WORKDIR $WDIR
RUN echo "\
JULIA_CPU_TARGET=arm1176jzf-s\n\
prefix=$WDIR/julia-$JL_VERSION\n\
LDFLAGS=-latomic\n\
USE_BINARYBUILDER = 0\n\
override USE_SYSTEM_BLAS=1\n\
override USE_SYSTEM_LAPACK=1\n\
override USE_SYSTEM_LIBM=1\n\
override USE_SYSTEM_GMP=1\n\
override USE_SYSTEM_MPFR=1\n\
" > Make.user && \
    cat Make.user && \
    git clone --depth=1 -b $JL_VERSION https://github.com/JuliaLang/julia.git $JL_BUILD_DIR &&\
    cp Make.user $JL_BUILD_DIR && \
    cd $JL_BUILD_DIR && make -j $(nproc) OPENBLAS_USE_THREAD=0 && make install

# add path of Julia
ENV PATH=$WDIR/julia-$JL_VERSION/bin:$PATH

# clean up
RUN rm -r $JL_BUILD_DIR $WDIR/Make.user
RUN apt-get clean && rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/*

Edit

I've added how to reproduce our issue

$ ls 
Dockerfile # see above
$ docker build -t rpizero .

[terasakisatoshi]

• I've tried to build again using gcc-8 to try whether new gcc will work for RPizero. But failed ... 😭

https://gist.github.com/terasakisatoshi/8771a7f846aea2f0f40db60a4802f329

# Build Julia binary for arm32-bit devices e.g. RaspberryPi Zero(W/WH)
FROM balenalib/raspberry-pi:buster-20191030

MAINTAINER SATOSHI TERASAKI

# install dependencies
RUN apt-get update && \
    apt-get install -y build-essential gcc-8 libatomic1 python gfortran-8 perl wget m4 cmake pkg-config git

# build julia from source
ARG JL_VERSION="v1.2.0"
ARG WDIR=/home/pi/work
ARG JL_BUILD_DIR=$WDIR/build
WORKDIR $WDIR
RUN echo "\
MARCH=armv6\n\
JULIA_CPU_TARGET=arm1176jzf-s\n\
USE_BINARYBUILDER = 0\n\
prefix=$WDIR/julia-$JL_VERSION\n\
LDFLAGS=-latomic\n\
override FC=gfortran-8\n\
override CC=gcc-8\n\
override CXX=g++-8\n\
" > Make.user && \
    cat Make.user && \
    git clone --depth=1 -b $JL_VERSION https://github.com/JuliaLang/julia.git $JL_BUILD_DIR &&\
    cp Make.user $JL_BUILD_DIR && \
    cd $JL_BUILD_DIR && make -j $(nproc) OPENBLAS_USE_THREAD=0 && make install

# add path of Julia
ENV PATH=$WDIR/julia-$JL_VERSION/bin:$PATH

# clean up
RUN rm -r $JL_BUILD_DIR $WDIR/Make.user
RUN apt-get clean && rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/*

[ViralBShah]

Someone familiar with llvm would need to dive in and figure out what is breaking.

[terasakisatoshi]

• I've found that setting USE_BINARYBUILDER=0 will solve issue for version==1.3 (and also yesterday's master) 👍 .
• Though I change several parameter (base image and build option), it does not solve for version=1.2 😭

# Build Julia binary for arm32-bit devices e.g. RaspberryPi Zero(W/WH)
FROM balenalib/raspberry-pi:buster-20191030

MAINTAINER SATOSHI TERASAKI

# install dependencies
RUN apt-get update && \
    apt-get install -y build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config \
    libopenblas-dev \
    liblapack-dev \
    libgmp3-dev \
    libmpfr-dev \
    git

# build julia from source
ARG JL_VERSION="v1.3.0"
ARG WDIR=/home/pi/work
ARG JL_BUILD_DIR=$WDIR/build
WORKDIR $WDIR
RUN echo "\
prefix=$WDIR/julia-$JL_VERSION\n\
JULIA_CPU_TARGET=arm1176jzf-s\n\
LDFLAGS=-latomic\n\
USE_BINARYBUILDER=0\n\
override USE_SYSTEM_BLAS=1\n\
override USE_SYSTEM_LAPACK=1\n\
override USE_SYSTEM_LIBM=1\n\
override USE_SYSTEM_GMP=1\n\
override USE_SYSTEM_MPFR=1\n\
" > Make.user && \
    cat Make.user && \
    git clone --depth=1 -b $JL_VERSION https://github.com/JuliaLang/julia.git $JL_BUILD_DIR &&\
    cp Make.user $JL_BUILD_DIR && \
    cd $JL_BUILD_DIR && make -j $(nproc) OPENBLAS_USE_THREAD=0 && make install

# add path of Julia
ENV PATH=$WDIR/julia-$JL_VERSION/bin:$PATH

# clean up
RUN rm -r $JL_BUILD_DIR $WDIR/Make.user
RUN apt-get clean && rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/*

[terasakisatoshi]

• I'll leave my prescription to solve our problem for those who would like to build Julia >= v.1.3.0 for Raspberry Pi (Zero). That is build docker image and extract julia binary from the container based on the docker image:

# Open terminal your work station
$ docker build -t julia4pi -f Dockerfile-v1.3.0 .
$ docker run --name jltmp julia4pi /bin/bash
$ docker cp jltmp:/home/pi/work/julia-v.1.3.0 /home/pi
$ ls
julia-v.1.3.0
$ scp -r julia-v1.3.0 pi@raspberrypi.local # this is your Raspberry Pi Zero

# Open terminal of your Raspberry Pi Zero e.g. ssh pi@raspberrypi.local
$ sudo apt-get update && \
    apt-get install -y build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config \
    libopenblas-dev \
    liblapack-dev \
    libgmp3-dev \
    libmpfr-dev
$ echo 'PATH=/home/pi/julia-v1.3.0:$PATH' >> /home/pi/.bashrc
$ source /home/pi/.bashrc
$ julia

• See my twitter to see the result(edit).

• I could confirm it works fine. However building Julia on Raspberry Pi zero locally fails.

[terasakisatoshi]

• I will close this issue via my prescription because

  • there is no reason to stick to Julia-v1.2.0. (Julia is going to update everytime-everyday)
  • I could confirm Julia-v.1.3.0 works on RPiZero.

• Much appreciated to @JeffBezanson giving a chance to submit the issue and @ViralBShah providing a advice USE_BINARYBUILDER = 0.