spack / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
4.29k stars 2.27k forks source link

Trilinos does not build anymore on OSX with Openblas (missing symbols) #736

Closed davydden closed 8 years ago

davydden commented 8 years ago

I gave it a quick try while working on PR and noticed that Trilinos (as of f5a77d39580d215f8d4948b2cbe6c8d47d4fd514) does not build on OSX due to missing blas/lapack symbols, see below. I am quite certain that it is related to recent changes in openblas https://github.com/LLNL/spack/commit/66bb71534b3ab7eaf7b27743efc0ecf9763e4982 . Possibly usage of this

 'CC=%s' % spack_cc,
 'FC=%s' % spack_fc

instead of

'CC=cc', 'FC=f77'

Otherwise I do not see anything that could make a noticeable difference.

Carefull examination confirms the missing symbols in openblas: current version:

$ nm -g libblas.dylib | grep _dgbcon_
$ nm -g liblapack.dylib | grep _dgbcon_

previous version:

$ nm -g libblas.dylib | grep _dgbcon_
00000000010a32f0 T _LAPACKE_dgbcon_work
0000000000a545de T _dgbcon_
$ nm -g libblas.dylib | grep _dgbcon_
00000000010a32f0 T _LAPACKE_dgbcon_work
0000000000a545de T _dgbcon_

The issue could be related to compiler wrappers and maybe somehow part of the openblas was not actually build. I must say that Trilinos builds on Ubuntu 14.04 with the current version of openblas.

This brings us back to the discussion of checking symbols. I think we should take from somewhere (even from the errors below for starters) the list of blas/lapack symbols and do sanity checks within each package that implements blas/lapack. Perhaps also make Travis build modified packages in each PR so that those type of issues appear sooner.

p.s. Actual linking error in Trilinos:

[ 18%] Linking CXX shared library libteuchosnumerics.dylib
Undefined symbols for architecture x86_64:
  "_dgbcon_", referenced from:
      Teuchos::LAPACK<int, double>::GBCON(char, int, int, int, double const*, int, int*, double, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgbequ_", referenced from:
      Teuchos::LAPACK<int, double>::GBEQU(int, int, int, int, double const*, int, double*, double*, double*, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgbrfs_", referenced from:
      Teuchos::LAPACK<int, double>::GBRFS(char, int, int, int, int, double const*, int, double const*, int, int const*, double const*, int, double*, int, double*, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgbtrf_", referenced from:
      Teuchos::LAPACK<int, double>::GBTRF(int, int, int, int, double*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgbtrs_", referenced from:
      Teuchos::LAPACK<int, double>::GBTRS(char, int, int, int, int, double const*, int, int const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgebak_", referenced from:
      Teuchos::LAPACK<int, double>::GEBAK(char, char, int, int, int, double const*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgebal_", referenced from:
      Teuchos::LAPACK<int, double>::GEBAL(char, int, double*, int, int, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgecon_", referenced from:
      Teuchos::LAPACK<int, double>::GECON(char, int, double const*, int, double, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgeequ_", referenced from:
      Teuchos::LAPACK<int, double>::GEEQU(int, int, double const*, int, double*, double*, double*, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgees_", referenced from:
      Teuchos::LAPACK<int, double>::GEES(char, char, int (*)(double*, double*), int, double*, int, int*, double*, double*, double*, int, double*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::GEES(char, int, double*, int, int*, double*, double*, double*, int, double*, int, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgeev_", referenced from:
      Teuchos::LAPACK<int, double>::GEEV(char, char, int, double*, int, double*, double*, double*, int, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::GEEV(char, char, int, double*, int, double*, double*, double*, int, double*, int, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgeevx_", referenced from:
      Teuchos::LAPACK<int, double>::GEEVX(char, char, char, char, int, double*, int, double*, double*, double*, int, double*, int, int*, int*, double*, double*, double*, double*, double*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgehrd_", referenced from:
      Teuchos::LAPACK<int, double>::GEHRD(int, int, int, double*, int, double*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgels_", referenced from:
      Teuchos::LAPACK<int, double>::GELS(char, int, int, int, double*, int, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgelss_", referenced from:
      Teuchos::LAPACK<int, double>::GELSS(int, int, int, double*, int, double*, int, double*, double, int*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::GELSS(int, int, int, double*, int, double*, int, double*, double, int*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgeqp3_", referenced from:
      Teuchos::LAPACK<int, double>::GEQP3(int, int, double*, int, int*, double*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgeqr2_", referenced from:
      Teuchos::LAPACK<int, double>::GEQR2(int, int, double*, int, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgeqrf_", referenced from:
      Teuchos::LAPACK<int, double>::GEQRF(int, int, double*, int, double*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgerfs_", referenced from:
      Teuchos::LAPACK<int, double>::GERFS(char, int, int, double const*, int, double const*, int, int const*, double const*, int, double*, int, double*, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgesv_", referenced from:
      Teuchos::LAPACK<int, double>::GESV(int, int, double*, int, int*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgesvd_", referenced from:
      Teuchos::LAPACK<int, double>::GESVD(char, char, int, int, double*, int, double*, double*, int, double*, int, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgesvx_", referenced from:
      Teuchos::LAPACK<int, double>::GESVX(char, char, int, int, double*, int, double*, int, int*, char, double*, double*, double*, int, double*, int, double*, double*, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgetrf_", referenced from:
      Teuchos::LAPACK<int, double>::GETRF(int, int, double*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgetri_", referenced from:
      Teuchos::LAPACK<int, double>::GETRI(int, double*, int, int const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgetrs_", referenced from:
      Teuchos::LAPACK<int, double>::GETRS(char, int, int, double const*, int, int const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgges_", referenced from:
      Teuchos::LAPACK<int, double>::GGES(char, char, char, int (*)(double*, double*, double*), int, double*, int, double*, int, int*, double*, double*, double*, double*, int, double*, int, double*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dggev_", referenced from:
      Teuchos::LAPACK<int, double>::GGEV(char, char, int, double*, int, double*, int, double*, double*, double*, double*, int, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dggevx_", referenced from:
      Teuchos::LAPACK<int, double>::GGEVX(char, char, char, char, int, double*, int, double*, int, double*, double*, double*, double*, int, double*, int, int*, int*, double*, double*, double*, double*, double*, double*, double*, int, int*, int*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::GGEVX(char, char, char, char, int, double*, int, double*, int, double*, double*, double*, double*, int, double*, int, int*, int*, double*, double*, double*, double*, double*, double*, double*, int, double*, int*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgglse_", referenced from:
      Teuchos::LAPACK<int, double>::GGLSE(int, int, int, double*, int, double*, int, double*, double*, double*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dgttrf_", referenced from:
      Teuchos::LAPACK<int, double>::GTTRF(int, double*, double*, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dgttrs_", referenced from:
      Teuchos::LAPACK<int, double>::GTTRS(char, int, int, double const*, double const*, double const*, double const*, int const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dhseqr_", referenced from:
      Teuchos::LAPACK<int, double>::HSEQR(char, char, int, int, int, double*, int, double*, double*, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dlamch_", referenced from:
      Teuchos::LAPACK<int, double>::LAMCH(char) const in Teuchos_LAPACK.cpp.o
  "_dlangb_", referenced from:
      Teuchos::LAPACK<int, double>::LANGB(char, int, int, int, double const*, int, double*) const in Teuchos_LAPACK.cpp.o
  "_dlapy2_", referenced from:
      Teuchos::LAPACK<int, double>::LAPY2(double, double) const in Teuchos_LAPACK.cpp.o
  "_dlarfg_", referenced from:
      Teuchos::LAPACK<int, double>::LARFG(int, double*, double*, int, double*) const in Teuchos_LAPACK.cpp.o
  "_dlarnv_", referenced from:
      Teuchos::LAPACK<int, double>::LARNV(int, int*, int, double*) const in Teuchos_LAPACK.cpp.o
  "_dlartg_", referenced from:
      Teuchos::LAPACK<int, double>::LARTG(double, double, double*, double*, double*) const in Teuchos_LAPACK.cpp.o
  "_dlascl_", referenced from:
      Teuchos::LAPACK<int, double>::LASCL(char, int, int, double, double, int, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dlaswp_", referenced from:
      Teuchos::LAPACK<int, double>::LASWP(int, double*, int, int, int, int const*, int) const in Teuchos_LAPACK.cpp.o
  "_dlatrs_", referenced from:
      Teuchos::LAPACK<int, double>::LATRS(char, char, char, char, int, double*, int, double*, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dorghr_", referenced from:
      Teuchos::LAPACK<int, double>::ORGHR(int, int, int, double*, int, double const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dorgqr_", referenced from:
      Teuchos::LAPACK<int, double>::ORGQR(int, int, int, double*, int, double const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::UNGQR(int, int, int, double*, int, double const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dorm2r_", referenced from:
      Teuchos::LAPACK<int, double>::ORM2R(char, char, int, int, int, double const*, int, double const*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::UNM2R(char, char, int, int, int, double const*, int, double const*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dormhr_", referenced from:
      Teuchos::LAPACK<int, double>::ORMHR(char, char, int, int, int, int, double const*, int, double const*, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dormqr_", referenced from:
      Teuchos::LAPACK<int, double>::ORMQR(char, char, int, int, int, double*, int, double const*, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::UNMQR(char, char, int, int, int, double*, int, double const*, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dpocon_", referenced from:
      Teuchos::LAPACK<int, double>::POCON(char, int, double const*, int, double, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dpoequ_", referenced from:
      Teuchos::LAPACK<int, double>::POEQU(int, double const*, int, double*, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dporfs_", referenced from:
      Teuchos::LAPACK<int, double>::PORFS(char, int, int, double*, int, double const*, int, double const*, int, double*, int, double*, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dposv_", referenced from:
      Teuchos::LAPACK<int, double>::POSV(char, int, int, double*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dposvx_", referenced from:
      Teuchos::LAPACK<int, double>::POSVX(char, char, int, int, double*, int, double*, int, char, double*, double*, int, double*, int, double*, double*, double*, double*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_dpotrf_", referenced from:
      Teuchos::LAPACK<int, double>::POTRF(char, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dpotri_", referenced from:
      Teuchos::LAPACK<int, double>::POTRI(char, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dpotrs_", referenced from:
      Teuchos::LAPACK<int, double>::POTRS(char, int, int, double const*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dpttrf_", referenced from:
      Teuchos::LAPACK<int, double>::PTTRF(int, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dpttrs_", referenced from:
      Teuchos::LAPACK<int, double>::PTTRS(int, int, double const*, double const*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dspev_", referenced from:
      Teuchos::LAPACK<int, double>::SPEV(char, char, int, double*, double*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dsteqr_", referenced from:
      Teuchos::LAPACK<int, double>::STEQR(char, int, double*, double*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dsyev_", referenced from:
      Teuchos::LAPACK<int, double>::SYEV(char, char, int, double*, int, double*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::HEEV(char, char, int, double*, int, double*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dsygv_", referenced from:
      Teuchos::LAPACK<int, double>::SYGV(int, char, char, int, double*, int, double*, int, double*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::HEGV(int, char, char, int, double*, int, double*, int, double*, double*, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dsytrd_", referenced from:
      Teuchos::LAPACK<int, double>::SYTRD(char, int, double*, int, double*, double*, double*, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dtgevc_", referenced from:
      Teuchos::LAPACK<int, double>::TGEVC(char, char, int const*, int, double*, int, double*, int, double*, int, double*, int, int, int*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dtgsen_", referenced from:
      Teuchos::LAPACK<int, double>::TGSEN(int, int, int, int const*, int, double*, int, double*, int, double*, double*, double*, double*, int, double*, int, int*, double*, double*, double*, double*, int, int*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dtrevc_", referenced from:
      Teuchos::LAPACK<int, double>::TREVC(char, char, int*, int, double const*, int, double*, int, double*, int, int, int*, double*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::TREVC(char, int, double const*, int, double*, int, double*, int, int, int*, double*, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dtrexc_", referenced from:
      Teuchos::LAPACK<int, double>::TREXC(char, int, double*, int, double*, int, int, int, double*, int*) const in Teuchos_LAPACK.cpp.o
  "_dtrsen_", referenced from:
      Teuchos::LAPACK<int, double>::TRSEN(char, char, int const*, int, double*, int, double*, int, double*, double*, int*, double*, double*, double*, int, int*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dtrtri_", referenced from:
      Teuchos::LAPACK<int, double>::TRTRI(char, char, int, double const*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_dtrtrs_", referenced from:
      Teuchos::LAPACK<int, double>::TRTRS(char, char, char, int, int, double const*, int, double*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_ilaenv_", referenced from:
      Teuchos::LAPACK<int, float>::ILAENV(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, int, int) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, double>::ILAENV(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, int, int) const in Teuchos_LAPACK.cpp.o
  "_sgbcon_", referenced from:
      Teuchos::LAPACK<int, float>::GBCON(char, int, int, int, float const*, int, int*, float, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgbequ_", referenced from:
      Teuchos::LAPACK<int, float>::GBEQU(int, int, int, int, float const*, int, float*, float*, float*, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgbrfs_", referenced from:
      Teuchos::LAPACK<int, float>::GBRFS(char, int, int, int, int, float const*, int, float const*, int, int const*, float const*, int, float*, int, float*, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgbtrf_", referenced from:
      Teuchos::LAPACK<int, float>::GBTRF(int, int, int, int, float*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgbtrs_", referenced from:
      Teuchos::LAPACK<int, float>::GBTRS(char, int, int, int, int, float const*, int, int const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgebak_", referenced from:
      Teuchos::LAPACK<int, float>::GEBAK(char, char, int, int, int, float const*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgebal_", referenced from:
      Teuchos::LAPACK<int, float>::GEBAL(char, int, float*, int, int, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgecon_", referenced from:
      Teuchos::LAPACK<int, float>::GECON(char, int, float const*, int, float, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgeequ_", referenced from:
      Teuchos::LAPACK<int, float>::GEEQU(int, int, float const*, int, float*, float*, float*, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgees_", referenced from:
      Teuchos::LAPACK<int, float>::GEES(char, char, int (*)(float*, float*), int, float*, int, int*, float*, float*, float*, int, float*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::GEES(char, int, float*, int, int*, float*, float*, float*, int, float*, int, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgeev_", referenced from:
      Teuchos::LAPACK<int, float>::GEEV(char, char, int, float*, int, float*, float*, float*, int, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::GEEV(char, char, int, float*, int, float*, float*, float*, int, float*, int, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgeevx_", referenced from:
      Teuchos::LAPACK<int, float>::GEEVX(char, char, char, char, int, float*, int, float*, float*, float*, int, float*, int, int*, int*, float*, float*, float*, float*, float*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgehrd_", referenced from:
      Teuchos::LAPACK<int, float>::GEHRD(int, int, int, float*, int, float*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgels_", referenced from:
      Teuchos::LAPACK<int, float>::GELS(char, int, int, int, float*, int, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgelss_", referenced from:
      Teuchos::LAPACK<int, float>::GELSS(int, int, int, float*, int, float*, int, float*, float, int*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::GELSS(int, int, int, float*, int, float*, int, float*, float, int*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgeqp3_", referenced from:
      Teuchos::LAPACK<int, float>::GEQP3(int, int, float*, int, int*, float*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgeqr2_", referenced from:
      Teuchos::LAPACK<int, float>::GEQR2(int, int, float*, int, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgeqrf_", referenced from:
      Teuchos::LAPACK<int, float>::GEQRF(int, int, float*, int, float*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgerfs_", referenced from:
      Teuchos::LAPACK<int, float>::GERFS(char, int, int, float const*, int, float const*, int, int const*, float const*, int, float*, int, float*, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgesv_", referenced from:
      Teuchos::LAPACK<int, float>::GESV(int, int, float*, int, int*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgesvd_", referenced from:
      Teuchos::LAPACK<int, float>::GESVD(char, char, int, int, float*, int, float*, float*, int, float*, int, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgesvx_", referenced from:
      Teuchos::LAPACK<int, float>::GESVX(char, char, int, int, float*, int, float*, int, int*, char, float*, float*, float*, int, float*, int, float*, float*, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgetrf_", referenced from:
      Teuchos::LAPACK<int, float>::GETRF(int, int, float*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgetri_", referenced from:
      Teuchos::LAPACK<int, float>::GETRI(int, float*, int, int const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgetrs_", referenced from:
      Teuchos::LAPACK<int, float>::GETRS(char, int, int, float const*, int, int const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgges_", referenced from:
      Teuchos::LAPACK<int, float>::GGES(char, char, char, int (*)(float*, float*, float*), int, float*, int, float*, int, int*, float*, float*, float*, float*, int, float*, int, float*, int, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sggev_", referenced from:
      Teuchos::LAPACK<int, float>::GGEV(char, char, int, float*, int, float*, int, float*, float*, float*, float*, int, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sggevx_", referenced from:
      Teuchos::LAPACK<int, float>::GGEVX(char, char, char, char, int, float*, int, float*, int, float*, float*, float*, float*, int, float*, int, int*, int*, float*, float*, float*, float*, float*, float*, float*, int, int*, int*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::GGEVX(char, char, char, char, int, float*, int, float*, int, float*, float*, float*, float*, int, float*, int, int*, int*, float*, float*, float*, float*, float*, float*, float*, int, float*, int*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgglse_", referenced from:
      Teuchos::LAPACK<int, float>::GGLSE(int, int, int, float*, int, float*, int, float*, float*, float*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sgttrf_", referenced from:
      Teuchos::LAPACK<int, float>::GTTRF(int, float*, float*, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sgttrs_", referenced from:
      Teuchos::LAPACK<int, float>::GTTRS(char, int, int, float const*, float const*, float const*, float const*, int const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_shseqr_", referenced from:
      Teuchos::LAPACK<int, float>::HSEQR(char, char, int, int, int, float*, int, float*, float*, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_slamch_", referenced from:
      Teuchos::LAPACK<int, float>::LAMCH(char) const in Teuchos_LAPACK.cpp.o
  "_slangb_", referenced from:
      Teuchos::LAPACK<int, float>::LANGB(char, int, int, int, float const*, int, float*) const in Teuchos_LAPACK.cpp.o
  "_slarfg_", referenced from:
      Teuchos::LAPACK<int, float>::LARFG(int, float*, float*, int, float*) const in Teuchos_LAPACK.cpp.o
  "_slarnv_", referenced from:
      Teuchos::LAPACK<int, float>::LARNV(int, int*, int, float*) const in Teuchos_LAPACK.cpp.o
  "_slartg_", referenced from:
      Teuchos::LAPACK<int, float>::LARTG(float, float, float*, float*, float*) const in Teuchos_LAPACK.cpp.o
  "_slascl_", referenced from:
      Teuchos::LAPACK<int, float>::LASCL(char, int, int, float, float, int, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_slaswp_", referenced from:
      Teuchos::LAPACK<int, float>::LASWP(int, float*, int, int, int, int const*, int) const in Teuchos_LAPACK.cpp.o
  "_slatrs_", referenced from:
      Teuchos::LAPACK<int, float>::LATRS(char, char, char, char, int, float*, int, float*, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sorghr_", referenced from:
      Teuchos::LAPACK<int, float>::ORGHR(int, int, int, float*, int, float const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sorgqr_", referenced from:
      Teuchos::LAPACK<int, float>::ORGQR(int, int, int, float*, int, float const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::UNGQR(int, int, int, float*, int, float const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sorm2r_", referenced from:
      Teuchos::LAPACK<int, float>::ORM2R(char, char, int, int, int, float const*, int, float const*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::UNM2R(char, char, int, int, int, float const*, int, float const*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sormhr_", referenced from:
      Teuchos::LAPACK<int, float>::ORMHR(char, char, int, int, int, int, float const*, int, float const*, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sormqr_", referenced from:
      Teuchos::LAPACK<int, float>::ORMQR(char, char, int, int, int, float*, int, float const*, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::UNMQR(char, char, int, int, int, float*, int, float const*, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_spocon_", referenced from:
      Teuchos::LAPACK<int, float>::POCON(char, int, float const*, int, float, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_spoequ_", referenced from:
      Teuchos::LAPACK<int, float>::POEQU(int, float const*, int, float*, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_sporfs_", referenced from:
      Teuchos::LAPACK<int, float>::PORFS(char, int, int, float*, int, float const*, int, float const*, int, float*, int, float*, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_sposv_", referenced from:
      Teuchos::LAPACK<int, float>::POSV(char, int, int, float*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sposvx_", referenced from:
      Teuchos::LAPACK<int, float>::POSVX(char, char, int, int, float*, int, float*, int, char, float*, float*, int, float*, int, float*, float*, float*, float*, int*, int*) const in Teuchos_LAPACK.cpp.o
  "_spotrf_", referenced from:
      Teuchos::LAPACK<int, float>::POTRF(char, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_spotri_", referenced from:
      Teuchos::LAPACK<int, float>::POTRI(char, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_spotrs_", referenced from:
      Teuchos::LAPACK<int, float>::POTRS(char, int, int, float const*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_spttrf_", referenced from:
      Teuchos::LAPACK<int, float>::PTTRF(int, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_spttrs_", referenced from:
      Teuchos::LAPACK<int, float>::PTTRS(int, int, float const*, float const*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_sspev_", referenced from:
      Teuchos::LAPACK<int, float>::SPEV(char, char, int, float*, float*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_ssteqr_", referenced from:
      Teuchos::LAPACK<int, float>::STEQR(char, int, float*, float*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_ssyev_", referenced from:
      Teuchos::LAPACK<int, float>::SYEV(char, char, int, float*, int, float*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::HEEV(char, char, int, float*, int, float*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_ssygv_", referenced from:
      Teuchos::LAPACK<int, float>::SYGV(int, char, char, int, float*, int, float*, int, float*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::HEGV(int, char, char, int, float*, int, float*, int, float*, float*, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_ssytrd_", referenced from:
      Teuchos::LAPACK<int, float>::SYTRD(char, int, float*, int, float*, float*, float*, float*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_stgevc_", referenced from:
      Teuchos::LAPACK<int, float>::TGEVC(char, char, int const*, int, float*, int, float*, int, float*, int, float*, int, int, int*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_stgsen_", referenced from:
      Teuchos::LAPACK<int, float>::TGSEN(int, int, int, int const*, int, float*, int, float*, int, float*, float*, float*, float*, int, float*, int, int*, float*, float*, float*, float*, int, int*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_strevc_", referenced from:
      Teuchos::LAPACK<int, float>::TREVC(char, char, int*, int, float const*, int, float*, int, float*, int, int, int*, float*, int*) const in Teuchos_LAPACK.cpp.o
      Teuchos::LAPACK<int, float>::TREVC(char, int, float const*, int, float*, int, float*, int, int, int*, float*, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_strexc_", referenced from:
      Teuchos::LAPACK<int, float>::TREXC(char, int, float*, int, float*, int, int, int, float*, int*) const in Teuchos_LAPACK.cpp.o
  "_strsen_", referenced from:
      Teuchos::LAPACK<int, float>::TRSEN(char, char, int const*, int, float*, int, float*, int, float*, float*, int*, float*, float*, float*, int, int*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_strtri_", referenced from:
      Teuchos::LAPACK<int, float>::TRTRI(char, char, int, float const*, int, int*) const in Teuchos_LAPACK.cpp.o
  "_strtrs_", referenced from:
      Teuchos::LAPACK<int, float>::TRTRS(char, char, char, int, int, float const*, int, float*, int, int*) const in Teuchos_LAPACK.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
davydden commented 8 years ago

indeed,

$ git diff
diff --git a/var/spack/repos/builtin/packages/openblas/package.py b/var/spack/repos/builtin/packages/openblas/package.py
index 4ec829a..39c49db 100644
--- a/var/spack/repos/builtin/packages/openblas/package.py
+++ b/var/spack/repos/builtin/packages/openblas/package.py
@@ -19,8 +19,8 @@ class Openblas(Package):

     def install(self, spec, prefix):
-        make_defs = ['CC=%s' % spack_cc,
-                     'FC=%s' % spack_fc]
+        make_defs = ['CC=cc',
+                     'FC=f77']

         make_targets = ['libs', 'netlib']

@@ -67,4 +67,3 @@ class Openblas(Package):
         if '+shared' in self.spec:
             self.spec.blas_shared_lib   = join_path(libdir, 'libopenblas.%s' % dso_suffix)
             self.spec.lapack_shared_lib = self.spec.blas_shared_lib
-

fixes missing symbols in Openblas (i have not tried to build Trilinos though). Note, that the build time changes from 1m 37.33s to 3m 42.35s with this change.

@tgamblin do you recall why you changed those arguments in Openblas? My understanding is that CC and F77 point to compiler wrappers, whereas spack_cc and spack_f77 are direct compilers. Why it makes that huge difference here -- i don't know.

alalazo commented 8 years ago

@davydden FC=f77 seems a bit suspicious...

tgamblin commented 8 years ago

@davydden: sorry I haven't gotten to this yet.

Both actually point to compiler wrappers. Hardcoding CC=cc and F77=f77 will always work in Spack; those are the generic compiler names. Originally, that is actually how we built most packages. spack_cc, etc. are absolute paths to the named links to the compiler wrappers. Here is the directory structure:

lib/spack/env/
    cc
    c++ -> cc
    f77 -> cc
    f90 -> cc
    gcc/
        gcc -> ../cc
        g++ .> ../cc
        gfortran -> ../cc

cc is just the name of the compiler wrapper; lib/spack/env is in the build PATH so invoking it will work. spack_cc is the absolute path to lib/spack/env/gcc/gcc, which also points to cc. The name matters for libtool and other builds that try to deduce the compiler from the name (FWIW, I'm curious whether this is going to help us on Cray, where the builds see a more informative name than they would compiling directly with the Cray cc/CC/ftn wrappers).

In the above example, the one thing I may have botched is changing f77 to spack_fc. That should probably be spack_f77. Do you mind trying that change out?

If that does not help, I think it means that the openblas build is doing something wrong w.r.t. assumptions about the compiler name on Mac OS X (clang? gcc?). openblas uses a pretty customized build so that is definitely possible... maybe cc and f77 are better for this case, but I wonder if that will cause it to pick the wrong linker flags elsewhere (on Linux).

davydden commented 8 years ago

@alalazo @tgamblin : here are some clues variant 1 (not working):

==> make -j8 CC=cc FC=fc libs netlib shared
OpenBLAS: Detecting fortran compiler failed. Cannot compile LAPACK. Only compile BLAS.
make[1]: warning: -jN forced in submake: disabling jobserver mode.

variant 2 (working):

==> make -j8 libs netlib shared CC=cc FC=f77
make[1]: warning: -jN forced in submake: disabling jobserver mode.

variant 3 (not working):

==> make -j8 CC=/Users/davydden/spack/lib/spack/env/clang/clang FC=/Users/davydden/spack/lib/spack/env/f90 libs netlib shared
OpenBLAS: Detecting fortran compiler failed. Cannot compile LAPACK. Only compile BLAS.
make[1]: warning: -jN forced in submake: disabling jobserver mode.

so it boils down to understand how exactly openblas detects fortran compiler and why does it fail in variant 3 and variant 1 but works in variant 2.

davydden commented 8 years ago

p.s. spack_f77 does not work either: *variant 4 (not working):

==> make -j8 CC=/Users/davydden/spack/lib/spack/env/clang/clang FC=/Users/davydden/spack/lib/spack/env/f90 libs netlib shared
OpenBLAS: Detecting fortran compiler failed. Cannot compile LAPACK. Only compile BLAS.
make[1]: warning: -jN forced in submake: disabling jobserver mode.
davydden commented 8 years ago

i thinks it is related to wrappers somehow... The same kind of hidden issues we had when CMake couldn't pick up working Fortran compiler for mixed C/Fortran projects: https://github.com/LLNL/spack/issues/634

tgamblin commented 8 years ago

@davydden: 4 looks the same as 3 to me. It says f90, not f77. Can you see if that one is correct? I suspect that one should work.

davydden commented 8 years ago

@tgamblin : you are right, i must have copy-pasted it from the wrong file

variant 4 (not working):

==> make -j8 CC=/Users/davydden/spack/lib/spack/env/clang/clang FC=/Users/davydden/spack/lib/spack/env/f77 libs netlib shared
OpenBLAS: Detecting fortran compiler failed. Cannot compile LAPACK. Only compile BLAS.
tgamblin commented 8 years ago

It looks like the build doesn't properly take the base name of the compiler to identify it. If I run the build with an absolute path to f77 it fails; if I run it with just f77 it succeeds. I tried this with this environment:

    $ spack env openblas%clang bash
    $ make CC=... FC=...

I think we want CC to be called clang, or the build won't get clang flags right. For f77 I can't do much, as compiler mixing isn't well supported right now so we use the generic name when compiling with clang.

Can you try making openblas/package.py set the compilers to point to os.path.basename(spack_cc) and os.path.basename(spack_f77)? That will get you the proper names without reverting entirely to the generically named wrappers.

FYI: I switched to using absolute compiler paths when I discovered it can really affect build time. Using an abspath avoids searching PATH entries on machines where compilers are in slow shared filesystems, and it sped up builds by a surprising amount. Most builds handle abspaths fine, but openblas is a roll-your-own build. I am pretty sure I like the speed enough NOT to switch back to just using names, but if a lot of builds behave this way it might be good to make it easier to use just the compiler names.

davydden commented 8 years ago

@tgamblin : this

$ git diff
diff --git a/var/spack/repos/builtin/packages/openblas/package.py b/var/spack/repos/builtin/packages/openblas/package.py
index 4ec829a..7405de3 100644
--- a/var/spack/repos/builtin/packages/openblas/package.py
+++ b/var/spack/repos/builtin/packages/openblas/package.py
@@ -19,8 +19,8 @@ class Openblas(Package):

     def install(self, spec, prefix):
-        make_defs = ['CC=%s' % spack_cc,
-                     'FC=%s' % spack_fc]
+        make_defs = ['CC=%s' % os.path.basename(spack_cc),
+                     'FC=%s' % os.path.basename(spack_f77)]

         make_targets = ['libs', 'netlib']

@@ -67,4 +67,3 @@ class Openblas(Package):
         if '+shared' in self.spec:
             self.spec.blas_shared_lib   = join_path(libdir, 'libopenblas.%s' % dso_suffix)
             self.spec.lapack_shared_lib = self.spec.blas_shared_lib
-

worked

==> make -j8 CC=clang FC=f77 libs netlib shared
make[1]: warning: -jN forced in submake: disabling jobserver mode.

I can create a PR if that's what you meant.

tgamblin commented 8 years ago

yes please! And thanks for helping to diagnose this!