search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.92k stars 5.49k forks source link

fftw fails to compile #551

Closed mjakl closed 12 years ago

mjakl commented 12 years ago

When compiling a recent Julia snapshot, I get this error:

libtool: link: gcc -shared -Wl,--whole-archive kernel/.libs/libkernel.a dft/.libs/libdft.a dft/scalar/.libs/libdft_scalar.a dft/scalar/codelets/.libs/libdft_scalar_codelets.a rdft/.libs/librdft.a rdft/scalar/.libs/librdft_scalar.a rdft/scalar/r2cf/.libs/librdft_scalar_r2cf.a rdft/scalar/r2cb/.libs/librdft_scalar_r2cb.a rdft/scalar/r2r/.libs/librdft_scalar_r2r.a reodft/.libs/libreodft.a api/.libs/libapi.a simd-support/.libs/libsimd_support.a simd-support/.libs/libsimd_sse2_nonportable.a dft/simd/sse2/.libs/libdft_sse2_codelets.a rdft/simd/sse2/.libs/librdft_sse2_codelets.a -Wl,--no-whole-archive -lm -Wl,-rpath -Wl,-soname -Wl,libfftw3f.so.3 -o .libs/libfftw3f.so.3.3.0 /usr/bin/ld: cannot find libfftw3f.so.3: No such file or directory collect2: ld returned 1 exit status make[3]: * [libfftw3f.la] Error 1 make[2]: * [install-recursive] Error 1 make[1]: * [/home/JuliaLang/src/julia-build/external/root/lib/libfftw3f.3.so] Error 2 make: * [julia-release] Error 2

Would it be possible to include a USE_SYSTEM_FFTW for distributions having this problem?

Please also see https://aur.archlinux.org/packages.php?ID=56877

ViralBShah commented 12 years ago

Yes, we should have a USE_SYSTEM_FFTW. I'll take a crack at it unless someone beats me to it. :-)

ViralBShah commented 12 years ago

Fixed in commit 0d2b254be191106b6e87348a28592e97ce046791. This was easy. Can you try it out?

mjakl commented 12 years ago

FFTW is working fine now. Thank you. Now gmp is having issues, but it seems to be an issues related to Archlinux' build system since issuing the (make) commands directly on the command line works fine. I'll have to dig into that.

ViralBShah commented 12 years ago

We should have a USE_SYSTEM_GMP as well.

-viral

On 10-Mar-2012, at 6:18 PM, Michael Jakl wrote:

FFTW is working fine now. Thank you. Now gmp is having issues, but it seems to be an issues related to Archlinux' build system since issuing the (make) commands directly on the command line works fine. I'll have to dig into that.

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4430780

mjakl commented 12 years ago

Unfortunately, the USE_SYSTEM_GMP switch does not work. The gmp-wrapper depends on the custom compiled gmp and starts the download/compilation (which in turn fails).

ViralBShah commented 12 years ago

Can you try now? Added a few more USESYSTEM* options also.

-viral

On 11-Mar-2012, at 4:39 AM, Michael Jakl wrote:

Unfortunately, the USE_SYSTEM_GMP switch does not work. The gmp-wrapper depends on the custom compiled gmp and starts the download/compilation (which in turn fails).

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4435170

mjakl commented 12 years ago

Thank you, the compilation phase works fine now. Yet, the install phase still searches for the libs and fails:

eposix.so lib/librandom.so lib/liblapack.so lib/libsuitesparse*so lib/libgrisu.so lib/libamos.so /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib
install: cannot stat `lib/libarpack.so': No such file or directory
`lib/libfdm.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libfdm.so'
install: cannot stat `lib/libfftw3.so': No such file or directory
install: cannot stat `lib/libfftw3f.so': No such file or directory
install: cannot stat `lib/libpcre.so': No such file or directory
install: cannot stat `lib/libpcrecpp.so': No such file or directory
install: cannot stat `lib/libpcreposix.so': No such file or directory
`lib/librandom.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/librandom.so'
install: cannot stat `lib/liblapack.so': No such file or directory
`lib/libsuitesparse.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse.so'
`lib/libsuitesparse_wrapper.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse_wrapper.so'
`lib/libgrisu.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libgrisu.so'
`lib/libamos.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libamos.so'
make: *** [install] Error 1
==> ERROR: A failure occurred in package().
    Aborting.

I'm using: USE_SYSTEM_LLVM=1 \ USE_SYSTEM_READLINE=1 \ USE_SYSTEM_PCRE=1 \ USE_SYSTEM_GMP=1 \ USE_SYSTEM_FFTW=1 \ USE_SYSTEM_LIBUNWIND=1 \ USE_SYSTEM_LAPACK=1 \ USE_SYSTEM_ARPACK=1 \ USE_SYSTEM_BLAS=1

ViralBShah commented 12 years ago

Hmm, I don't think I have tested the installation ever with all these Makefile flags. This will take some work. However, you don't really need to use install if you just copy the relevant files, you should be good to go.

-viral

On 11-Mar-2012, at 4:06 PM, Michael Jakl wrote:

Thank you, the compilation phase works fine now. Yet, the install phase still searches for the libs and fails:

eposix.so lib/librandom.so lib/liblapack.so lib/libsuitesparse_so lib/libgrisu.so lib/libamos.so /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib install: cannot stat lib/libarpack.so': No such file or directory lib/libfdm.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libfdm.so' install: cannot statlib/libfftw3.so': No such file or directory install: cannot stat lib/libfftw3f.so': No such file or directory install: cannot statlib/libpcre.so': No such file or directory install: cannot stat lib/libpcrecpp.so': No such file or directory install: cannot statlib/libpcreposix.so': No such file or directory lib/librandom.so' ->/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/librandom.so' install: cannot stat lib/liblapack.so': No such file or directory lib/libsuitesparse.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse.so' lib/libsuitesparse_wrapper.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse_wrapper.so' lib/libgrisu.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libgrisu.so' lib/libamos.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libamos.so' make: _\ [install] Error 1 ==> ERROR: A failure occurred in package(). Aborting.

I'm using: USE_SYSTEM_LLVM=1 \ USE_SYSTEM_READLINE=1 \ USE_SYSTEM_PCRE=1 \ USE_SYSTEM_GMP=1 \ USE_SYSTEM_FFTW=1 \ USE_SYSTEM_LIBUNWIND=1 \ USE_SYSTEM_LAPACK=1 \ USE_SYSTEM_ARPACK=1 \ USE_SYSTEM_BLAS=1

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438088

mjakl commented 12 years ago

I'm trying to get my Archlinux package working again, hence all those USE_SYSTEM flags. As a last resort, I could just copy the files. I'll keep you posted. Thanks for your help!

mjakl commented 12 years ago

Replacing line 36 of julia's Makefiles with >-install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib works with all those switches.

We need all those libraries in the lib folder, don't we?

ViralBShah commented 12 years ago

The simplest thing I can think of is something like:

test -f lib/libfoo.$(SHLIB_EXT) && install -v ...

This doesn't work in the Makefile because for files that are not compiled, test returns an error and the make fails. We need a quick fix for this, and a long term fix for the whole external libraries thing.

-viral

On 11-Mar-2012, at 4:06 PM, Michael Jakl wrote:

Thank you, the compilation phase works fine now. Yet, the install phase still searches for the libs and fails:

eposix.so lib/librandom.so lib/liblapack.so lib/libsuitesparse_so lib/libgrisu.so lib/libamos.so /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib install: cannot stat lib/libarpack.so': No such file or directory lib/libfdm.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libfdm.so' install: cannot statlib/libfftw3.so': No such file or directory install: cannot stat lib/libfftw3f.so': No such file or directory install: cannot statlib/libpcre.so': No such file or directory install: cannot stat lib/libpcrecpp.so': No such file or directory install: cannot statlib/libpcreposix.so': No such file or directory lib/librandom.so' ->/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/librandom.so' install: cannot stat lib/liblapack.so': No such file or directory lib/libsuitesparse.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse.so' lib/libsuitesparse_wrapper.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse_wrapper.so' lib/libgrisu.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libgrisu.so' lib/libamos.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libamos.so' make: _\ [install] Error 1 ==> ERROR: A failure occurred in package(). Aborting.

I'm using: USE_SYSTEM_LLVM=1 \ USE_SYSTEM_READLINE=1 \ USE_SYSTEM_PCRE=1 \ USE_SYSTEM_GMP=1 \ USE_SYSTEM_FFTW=1 \ USE_SYSTEM_LIBUNWIND=1 \ USE_SYSTEM_LAPACK=1 \ USE_SYSTEM_ARPACK=1 \ USE_SYSTEM_BLAS=1

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438088

ViralBShah commented 12 years ago

Well - we can do without the LLVM libraries, but yes, no harm for the time being.

-viral

On 11-Mar-2012, at 5:24 PM, Michael Jakl wrote:

Replacing line 36 of julia's Makefiles with

> -install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib

Works with all those switches.

We need all those libraries in the lib folder, don't we?

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438503

ViralBShah commented 12 years ago

BTW, just out of curiosity, does Archlinux have lapack, arpack-ng, suitesparse etc? What USE_SYSTEM flags did you end up using? Currently, these are all motivated by what is available on Debian. The only ones not in debian are double-conversion, amos, fdlibm, and dsfmt.

-viral

On 11-Mar-2012, at 5:24 PM, Michael Jakl wrote:

Replacing line 36 of julia's Makefiles with

> -install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib

Works with all those switches.

We need all those libraries in the lib folder, don't we?

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438503

mjakl commented 12 years ago

I'm currently compiling using these flags:

    USE_SYSTEM_LLVM=1 \
    USE_SYSTEM_READLINE=1 \
    USE_SYSTEM_PCRE=1 \
    USE_SYSTEM_GMP=1 \
    USE_SYSTEM_LIBUNWIND=1 \
    USE_SYSTEM_FFTW=1 \
    USE_SYSTEM_LAPACK=1 \
    USE_SYSTEM_ARPACK=1 \
    USE_SYSTEM_BLAS=1

With these dependencies from Archlinux: depends=('llvm' 'readline' 'pcre' 'gcc-fortran' 'fftw' 'gmp' 'lapack' 'blas' 'arpack' 'libunwind')

The versions are currently:

SuiteSparse is also available via AUR, but did not compile on my machine, hence I'm using the one julia compiles itself.

Thank you very much for all your help. I've been able to update the julia-git package on AUR now (https://aur.archlinux.org/packages.php?ID=56877).

mjakl commented 12 years ago

I forgot, double-conversion, amos, fdlibm, and dsfmt aren't available on Archlinux too.

ViralBShah commented 12 years ago

BTW, looking at the archlinux blas package, it looks like it is using the reference BLAS. Is this the BLAS that julia will also ending up using?

http://www.archlinux.org/packages/extra/i686/blas/

If so, this is likely to be orders of magnitude slower than something like openblas. Unfortunately, currently the blas and lapack in julia are inseparable, and we need some cleanup in the way the libraries are built.

It's a good idea for archlinux to provide openblas, and for julia to depend on that instead at least for platforms that openblas supports. On others, ATLAS may still be faster than reference blas.

-viral

On 11-Mar-2012, at 6:14 PM, Michael Jakl wrote:

I'm currently compiling using these flags:

   USE_SYSTEM_LLVM=1 \
   USE_SYSTEM_READLINE=1 \
   USE_SYSTEM_PCRE=1 \
   USE_SYSTEM_GMP=1 \
   USE_SYSTEM_LIBUNWIND=1 \
   USE_SYSTEM_FFTW=1 \
   USE_SYSTEM_LAPACK=1 \
   USE_SYSTEM_ARPACK=1 \
   USE_SYSTEM_BLAS=1

With these dependencies from Archlinux: depends=('llvm' 'readline' 'pcre' 'gcc-fortran' 'fftw' 'gmp' 'lapack' 'blas' 'arpack' 'libunwind')

The versions are currently:

  • LLVM 3.0
  • READLINE 6.2.002
  • PCRE 8.30
  • GMP 5.0.4
  • FORTRAN 4.6.3
  • LIBUNWIND 1.0.1 (only available via the use repository - AUR)
  • FFTW 3.3
  • LAPACK 3.4.0
  • ARPACK 3.1.0
  • BLAS 3.4.0

SuiteSparse is also available via AUR, but did not compile on my machine, hence I'm using the one julia compiles itself.

Thank you very much for all your help. I've been able to update the julia-git package on AUR now (https://aur.archlinux.org/packages.php?ID=56877).

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438731

mjakl commented 12 years ago

I see. The BLAS package would indeed be the reference implementation. I've reverted back to using lapack and openblas provided by julia itself. Thanks again :)

ViralBShah commented 12 years ago

What happens with ARPACK? Would it link to the Archlinux ARPACK, LAPACK, and BLAS? Sorry, all the stuff that links to BLAS and LAPACK is a bit of a mess right now.

-viral

On 11-Mar-2012, at 6:47 PM, Michael Jakl wrote:

I see. The BLAS package would indeed be the reference implementation. I've reverted back to using lapack and openblas provided by julia itself. Thanks again :)

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438909

ViralBShah commented 12 years ago

It should be ok to use the system arpack, which uses the system lapack and blas for now, because arpack doesn't seem to use level 3 BLAS.

Eventually, though, it's best to have Archlinux include openblas.

-viral

On 11-Mar-2012, at 6:47 PM, Michael Jakl wrote:

I see. The BLAS package would indeed be the reference implementation. I've reverted back to using lapack and openblas provided by julia itself. Thanks again :)

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4438909

mjakl commented 12 years ago

I'd like to integrate julia nicely into the system, but it seems that I'd need to create my own set of arpack, lapack, openblas libs, I think it's OK to use the custom ones directly in julia where necessary.

ViralBShah commented 12 years ago

I think that is a good idea. We will have more packages that will depend on BLAS. CLP is in the works somewhere, and then there is SuiteSparse already that needs a good BLAS.

-viral

On 11-Mar-2012, at 7:42 PM, Michael Jakl wrote:

I see, I'll look into creating a fitting OpenBLAS package. In the meantime I'll leave arpack as system library.

Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/551#issuecomment-4439229