Building without downloading anything

[mboisson]

Hi, Could you document what versions of dependencies are needed, and how to point the installer to already-existing versions of each dependency ?

Thank you

[mboisson]

In particular, it seems that OpenBLAS is required. Can it be compiled with MKL instead, which is our preferred implementation of BLAS.

The cmake also downloads things without saying what it is :

 228 -- HDF5 headers : /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/hdf5/1.10.6/include
 229 -- [download 100% complete]
 230 -- [download 0% complete]
 231 -- [download 6% complete]
 232 -- [download 13% complete]
 233 -- [download 19% complete]
 234 -- [download 25% complete]
 235 -- [download 32% complete]
 236 -- [download 38% complete]
 237 -- [download 44% complete]
 238 -- [download 51% complete]
 239 -- [download 57% complete]
 240 -- [download 63% complete]
 241 -- [download 70% complete]
 242 -- [download 76% complete]
 243 -- [download 82% complete]
 244 -- [download 89% complete]
 245 -- [download 95% complete]
 246 -- [download 100% complete]
 247 -- [download 100% complete]
 248 -- Found BISON: /cvmfs/soft.computecanada.ca/gentoo/2020/usr/bin/bison (found version "3.1")

[gdevenyi]

Hi,

The superbuild versions are all inside: https://github.com/BIC-MNI/minc-toolkit-v2/tree/develop-1.9.19/cmake-modules

You can attempt the USE_SYSTEM_<PACKAGENAME> settings in cmake to point at pre-installed/compiled versions of the dependencies.

[mboisson]

I have already specified the parameters I could find through ccmake :

 22 configopts = '-DUSE_SYSTEM_FFTW3D=ON -DFFTW3D_DIR=$EBROOTFFTW '
 23 configopts += '-DUSE_SYSTEM_FFTW3F=ON '
 24 configopts += '-DOPENJPEG_INCLUDE_DIR=$EBROOTGENTOO/include/openjpeg-2.3 '
 25 configopts += '-DUSE_SYSTEM_GSL=ON '
 26 configopts += '-DUSE_SYSTEM_HDF5=ON '
 27 configopts += '-DUSE_SYSTEM_NETCDF=ON '
 28 configopts += '-DUSE_SYSTEM_PCRE=ON '
 29 configopts += '-DUSE_SYSTEM_ZLIB=ON '
 30 configopts += '-DUSE_SYSTEM_ITK=ON '
 31 preconfigopts = "LDFLAGS='-lhdf5' "

and yet, it still downloads stuff.

[gdevenyi]

The settings can be configured using ccmake interactively, where you can see all the settings and profide paths.

[gdevenyi]

Please open a bug for each USE_SYSTEM_<PACKAGE> which does not work as you expect.

[mboisson]

It would help figuring that out if cmake did tell me what it is downloading. It does not say anything like "did not find X, downloading X" ... I have no idea what it's downloading

[mboisson]

Is there no way to just tell it "don't download anything, just stop and give me an error if you can't find something" ?

[gdevenyi]

No.

This was never configured for the gentoo offline-only build system.

Here's what a default cmake downloads:

l15:39:53 $ ls -l cache/
total 63332
-rw-rw-r-- 1 gdevenyi gdevenyi 19109741 2021-01-11 15:39 InsightToolkit-4.13.3.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi   326782 2021-01-11 15:39 c3d-v0.0.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi  4110137 2021-01-11 15:39 fftw-3.3.8.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi  5916715 2021-01-11 15:38 gsl-2.4.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi  8813699 2021-01-11 15:38 hdf5-1.10.6.tar.bz2
-rw-rw-r-- 1 gdevenyi gdevenyi  2158825 2021-01-11 15:38 libjpeg-turbo-1.5.90.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi    49003 2021-01-11 15:38 liblbfgs-v1.10-cmake.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi 19711158 2021-01-11 15:38 netcdf-v4.7.4.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi   258491 2021-01-11 15:38 nifticlib-2.0.0.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi  2207329 2021-01-11 15:38 openjpeg-2.3.0.tar.gz
-rw-rw-r-- 1 gdevenyi gdevenyi  1560119 2021-01-11 15:38 pcre-8.40.tar.bz2
-rw-rw-r-- 1 gdevenyi gdevenyi   607698 2021-01-11 15:38 zlib-1.2.11.tar.gz

[gdevenyi]

The configuration does not currently have a setting for USE_SYSTEM_OPENBLAS

CMakeLists.txt:OPTION(USE_SYSTEM_ZLIB       "Use System ZLIB "       OFF)
CMakeLists.txt:#OPTION(USE_SYSTEM_NIFTI      "Use System NIFTI "       OFF)
CMakeLists.txt:#OPTION(USE_SYSTEM_GIFTI      "Use System GIfTI"       OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_NETCDF     "Use System NETCDF > 3.6.3" OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_HDF5       "Use System HDF5 > 1.8"     OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_PCRE       "Use System PCRE "        OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_GSL        "Use System GSL "         OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_FFTW3F     "Use System FFTW3F "      OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_FFTW3D     "Use System FFTW3D "      OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_GLUT       "Use System GLUT  "       ON)
CMakeLists.txt:OPTION(USE_SYSTEM_OPENJPEG   "Use System OpenJPEG  "   OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_JPEG       "Use System JPEG  "   OFF)
CMakeLists.txt:OPTION(USE_SYSTEM_ITK        "Use System ITK" OFF)

[mboisson]

For OpenBLAS, it would actually be best to allow using MKL. OpenBLAS's performance is lacking for AVX512 CPUs.

[vfonov]

I would suggest to use develop-1.9.18 , the develop-1.9.19 will get broken, once I merge the changes from 1.9.18

[mboisson]

I'm using releases, not develop.

[vfonov]

For OpenBLAS, it would actually be best to allow using MKL. OpenBLAS's performance is lacking for AVX512 CPUs

currently the OpenBLAS is used in a very limited number of places, so changing to MKL will not affect a lot. I would suggest to compile completely without OpenBLAS if there are some problems with compiling it.

[vfonov]

https://github.com/vfonov/conda-recipes/blob/develop/minc-toolkit-v2/build.sh - an example on how to build with using maximum number of pre-built libraries (it's a recipe for https://anaconda.org/vfonov/minc-toolkit-v2 )

[mboisson]

I think I managed to proceed to the build phase with minimal downloads. It now fails during the build, with missing headers. I opened a separate issue for it. I will close this one.