Feature/bundle request: Julia

[halars]

It would be useful to have Julia (https://julialang.org) available on Clear Linux as a bundle. It fits well within the scope of Clear Linux in my opinion.

The pre-compiled "Generic Linux Binaries" available (https://julialang.org/downloads), already run without any issues--that I have noticed--so any extreme work to implement it as a bundle, shouldn't be required.

The benefits would be easy deployment and updates, and potentially some performance optimisations.

[fenrus75]

I looked at Julia a year ago.... I’ll look again tomorrow

On Sun, Jul 22, 2018 at 18:58 Lars Hansson notifications@github.com wrote:

It would be useful to have Julia (https://julialang.org) available on Clear Linux as a bundle. It fits well within the scope of Clear Linux in my opinion.

The pre-compiled "Generic Linux Binaries" available ( https://julialang.org/downloads), already run without any issues--that I have noticed--so any extreme work to implement it as a bundle, shouldn't be required.

The benefits would be easy deployment and updates, and potentially some performance optimisations.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/clearlinux/distribution/issues/126, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPeFUnRsD9UuKvdMEFLzIS-KyTQei64ks5uJS3igaJpZM4VaQTP .

[fenrus75]

so far it is julia 4 - arjan 0

still poking at it

[iphutch]

@fenrus75 Any wins to your name in this space?

[rahulunair]

@iphutch @fenrus75 any update on this ?

[chriselrod]

I'm running into issues trying to build Julia from source on Clear Linux that I've never seen before:

gfortran: error: unrecognized command line option ‘-i8’; did you mean ‘-k8’?

I've never had problems building from source before on other distros (Ubuntu / Fedora / Arch / Antergos).

-fdefault-integer-8 is the correct flag for using 8-byte integers. The errors seem to come up while building OpenBLAS. The makefile checks:

ifeq ($(F_COMPILER), GFORTRAN)
CCOMMON_OPT += -DF_INTERFACE_GFORT
FCOMMON_OPT += -Wall
# make single-threaded LAPACK calls thread-safe #1847
FCOMMON_OPT += -frecursive
#Don't include -lgfortran, when NO_LAPACK=1 or lsbcc
ifneq ($(NO_LAPACK), 1)
EXTRALIB += -lgfortran
endif
ifdef NO_BINARY_MODE
ifeq ($(ARCH), $(filter $(ARCH),mips64))
ifdef BINARY64
FCOMMON_OPT += -mabi=64
else
FCOMMON_OPT += -mabi=n32
endif
else ifeq ($(ARCH), $(filter $(ARCH),mips))
FCOMMON_OPT += -mabi=32
endif
else
ifdef BINARY64
ifneq ($(OSNAME), AIX)
FCOMMON_OPT += -m64
endif
ifdef INTERFACE64
ifneq ($(INTERFACE64), 0)
FCOMMON_OPT +=  -fdefault-integer-8
endif
endif
else
ifneq ($(OSNAME), AIX)
FCOMMON_OPT += -m32
endif
endif
endif
ifeq ($(USE_OPENMP), 1)
FCOMMON_OPT += -fopenmp
endif
endif

ifeq ($(F_COMPILER), INTEL)
CCOMMON_OPT += -DF_INTERFACE_INTEL
ifdef INTERFACE64
ifneq ($(INTERFACE64), 0)
FCOMMON_OPT += -i8
endif
endif
ifeq ($(USE_OPENMP), 1)
FCOMMON_OPT += -fopenmp
endif
endif

Similarly, Julia's build system also checks:

USE_BLAS_FFLAGS :=
ifeq ($(USE_BLAS64), 1)
ifeq ($(USEIFC),1)
USE_BLAS_FFLAGS += -i8
else
USE_BLAS_FFLAGS += -fdefault-integer-8
endif
endif

So it seems like, somewhere, some build system is getting confused and accidentally passing '-i8' instead of '-fdefault-integer-8'. Unfortunately, Julia's package ecosystem assumes Julia has been built with 64-bit OpenBLAS. If it weren't for this, I'd build with either MKL or 32-bit OpenBLAS.

Any idea how I may be able to address this? Or why the build systems are confused?

EDIT:

$ make VERBOSE=1
...
...
gfortran -march=native -m64 -g -O3 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wl,--copy-dt-needed-entries -m64 -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -malign-data=abi -fno-semantic-interposition -ftree-vectorize  -ftree-loop-vectorize -Wl,--enable-new-dtags  -O2 -fPIC -O2 -i8 -fPIC -march=skylake-avx512 -c -o sbdsvdx.o sbdsvdx.f
gfortran: error: unrecognized command line option ‘-i8’; did you mean ‘-k8’?

checking lapack-netlib/make.inc in the OpenBLAS build directory, I see the flags:

OPTS        = -g -O3 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wl,--copy-dt-needed-entries -m64 -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -malign-data=abi -fno-semantic-interposition -ftree-vectorize  -ftree-loop-vectorize -Wl,--enable-new-dtags  -O2 -fPIC -O2 -i8 -fPIC -march=skylake-avx512
POPTS       = -i8 -fPIC -march=skylake-avx512 -pg -fno-inline
NOOPT       = -O0 -g -feliminate-unused-debug-types -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wl,--copy-dt-needed-entries -m64 -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -malign-data=abi -fno-semantic-interposition -ftree-vectorize -ftree-loop-vectorize -Wl,--enable-new-dtags -fPIC -i8 -fPIC -march=skylake-avx512
PNOOPT      = -i8 -fPIC -march=skylake-avx512 -pg -fno-inline -O0
LOADOPTS    = -g -O3 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wl,--copy-dt-needed-entries -m64 -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -malign-data=abi -fno-semantic-interposition -ftree-vectorize  -ftree-loop-vectorize -Wl,--enable-new-dtags  -O2 -fPIC -O2 -i8 -fPIC -march=skylake-avx512 -lm -lpthread

They all contain -8, and a whole lot of extra flags not present when building on other OSes. Here is where OpenBLAS seems to create that make.inc file.

I fixed the problem by modifying this line:

ifeq ($(F_COMPILER), INTEL)
CCOMMON_OPT += -DF_INTERFACE_INTEL
ifdef INTERFACE64
ifneq ($(INTERFACE64), 0)
FCOMMON_OPT += -fdefault-integer-8 # -i8
endif
endif
ifeq ($(USE_OPENMP), 1)
FCOMMON_OPT += -fopenmp
endif
endif

to my surprise, this let the compilation proceed until I hit a segmentation fault:

 Test of subprogram number  1            ZDOTC 

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f96fd3dea1f in ???
#1  0x4094a0 in zdotc_k
    at ../kernel/x86_64/zdot.c:140
#2  0x404d4c in zdotc_
    at /home/chriselrod/Documents/libraries/OpenBLAS/interface/zdot.c:123
#3  0x404092 in check2_
    at /home/chriselrod/Documents/libraries/OpenBLAS/test/zblat1.f:493
#4  0x40477c in zblat1
    at /home/chriselrod/Documents/libraries/OpenBLAS/test/zblat1.f:36
#5  0x40243e in main
    at /home/chriselrod/Documents/libraries/OpenBLAS/test/zblat1.f:47
dblat2.f:2003:0:

EDIT: passing F_COMPILER=GFORTRAN to make also fixed the problem.

But it is odd that without it, OpenBLAS somehow things the compiler is INTEL, even while using gfortran. Probably an OpenBLAS bug?

[iphutch]

@bryteise

[bryteise]

I am sadly not likely to get to this one soon but we definitely want to add it. As to the oddity @chriselrod hit I'd be a little surprised that it is an OpenBLAS bug though that is not a fun package to build. Could be just how julia is trying to build with it? As an aside you can see how Clear Linux is building OpenBLAS via: https://github.com/clearlinux-pkgs/openblas/blob/master/openblas.spec

[chriselrod]

Using "INTERFACE64=1" triggers the specific error I was getting, even when cloning OpenBLAS directly and trying to build it. The error was: gfortran: error: unrecognized command line option ‘-i8’; did you mean ‘-k8’? The error comes up because ifort uses -i8 for 64-bit integers, while gfortran uses -fdefault-integer-8, and the OpenBLAS build system gets confused into defining F_COMPILER=Intel.

The solution I used, which I see Clear Linux's build script was using too, was passing "F_COMPILER=GFORTRAN".

I don't know perl, but I think this code from the OpenBLAS build system is to blame. That is, it runs: gfortran -O2 -S ftest.f > /dev/null 2>&1 && cat ftest.s && rm -f ftest.s:

    .file   "ftest.f"
    .text
    .p2align 4,,15
    .globl  zhoge_
    .type   zhoge_, @function
zhoge_:
.LFB0:
    .cfi_startproc
    pxor    %xmm1, %xmm1
    pxor    %xmm0, %xmm0
    ret
    .cfi_endproc
.LFE0:
    .size   zhoge_, .-zhoge_
    .ident  "GCC: (Clear Linux OS for Intel Architecture) 8.2.1 20190208 gcc-8-branch@268661"
    .section    .note.GNU-stack,"",@progbits

and then I think it pattern matches the "Intel" at the end to identify the compiler as ifort. =/ Hence, the need for explicit "F_COMPILER=GFORTRAN". I could be totally wrong though.

[benjaminglass1]

It should be possible to override Julia's custom build of OpenBLAS and force it to use the system version using two make flags:

USE_SYSTEM_BLAS = 1
USE_SYSTEM_LAPACK = 1

This behavior is documented in Julia's Make.inc file (lines 43 and 44) and utilized in the Arch Linux build (lines 22-24) of Julia, for reference.

[chriselrod]

Recently I've been building Julia (on Clear Linux) with MKL. Before that, IIRC, I could build master (but not the 1.1 release) by simply adding F_COMPILER=GFORTRAN to OPENBLAS_BUILD_OPTS.

I don't know much about licensing. Eg, why Microsoft can ship R with MKL but folks at Julia say it violates the GPL. Julia + MKL should be faster than Julia + OpenBLAS on most any LinearAlgebra program on an Intel system -- especially those with avx512, where OpenBLAS's support isn't great yet.

[rana]

Julia now builds on Clear Linux with Julia version 1.2.0-rc1.0.

git clone git://github.com/JuliaLang/julia.git
cd julia
git checkout v1.2.0-rc1
make # or make -j 16
./julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.2.0-rc1.0 (2019-05-30)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> exit()

@fredrikekre Mentioned upcoming support for Julia on Alpine. So I figured I'd try building with that branch. Turns out it works.

So next steps.

How do we get an official Julia bundle? MKL support?

[bryteise]

Ah lovely, well we need to tweak julia's build system to not download anything which is not always super straight forward. @ahkok did you have some Julia progress that might be unblocked with the new rc?

[fenrus75]

julia builds. it just ends up breaking the whole rest of the distro ;-(

On Wed, Jun 19, 2019 at 7:16 AM William Douglas notifications@github.com wrote:

Ah lovely, well we need to tweak julia's build system to not download anything which is not always super straight forward. @ahkok https://github.com/ahkok did you have some Julia progress that might be unblocked with the new rc?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/clearlinux/distribution/issues/126?email_source=notifications&email_token=AAJ54FLJEXTKBYDWW24WUD3P3I5UFA5CNFSM4FLJATH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYCAMZY#issuecomment-503580263, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ54FLYZYW3Y5YLYUCGLGDP3I5UFANCNFSM4FLJATHQ .

[rana]

Which issues are breaking the distro? How to mitigate?

[srgk26]

Hi! So is there any update on this? It's not hard to install Julia myself, but it would be nice to have Julia in the official repo, especially one that is explicitly built with MKL. As for MKL, it might be worth checking out this though: https://github.com/JuliaComputing/MKL.jl. Though it's not the same as building using MKL from source, it's just 2-steps to switch from OpenBLAS to MKL.

[wpeguero]

Hello I would also be very interested in having Julia. Any update in this regard?

[chriselrod]

If you want to build Julia from source locally, it should be fine. BinaryBuilder takes care of most binaries needed, i.e. they'll just be downloaded automatically.

If you also want to build dependencies yourself, you'll need to modify the build script to get OpenBLAS to build as described above:

diff --git a/deps/openblas.mk b/deps/openblas.mk
index 50873c9220..5a50066619 100644
--- a/deps/openblas.mk
+++ b/deps/openblas.mk
@@ -5,7 +5,7 @@ OPENBLAS_GIT_URL := https://github.com/xianyi/OpenBLAS.git
 OPENBLAS_TAR_URL = https://api.github.com/repos/xianyi/OpenBLAS/tarball/$1
 $(eval $(call git-external,openblas,OPENBLAS,,,$(BUILDDIR)))

-OPENBLAS_BUILD_OPTS := CC="$(CC)" FC="$(FC)" LD="$(LD)" RANLIB="$(RANLIB)" TARGET=$(OPENBLAS_TARGET_ARCH) BINARY=$(BINARY)
+OPENBLAS_BUILD_OPTS := CC="$(CC)" FC="$(FC)" LD="$(LD)" RANLIB="$(RANLIB)" TARGET=$(OPENBLAS_TARGET_ARCH) BINARY=$(BINARY) F_COMPILER=GFORTRAN

 # Thread support
 ifeq ($(OPENBLAS_USE_THREAD), 1)

FWIW, if you have an Intel CPU and care about BLAS/LAPACK performance, I'd generally recommend using MKL.jl (e.g., I have using MKL in my ~/.julia/config/startup.jl) and not worry too much about OpenBLAS.

It is not recommended to use Julia with system libraries. That is important to consider for anyone trying to package Julia with their distro. It's much better to just download an official binary or build from source; Julias installed from distro repos are often broken in some way.

Also worth pointing out that Clear Linux is still on LLVM 11, while Julia's latest release is on LLVM 12.