Add galario to conda

[fredRos]

To simplify the installation for users, we can add galario to conda-forge now that we have v1.0 This place has all the links to get started https://github.com/conda-forge/staged-recipes

Basically we have to create a .yml file that wraps up how we want galario to be built, then there are binary packages for mac os and linux. I think we can have a fairly easy time since all our our dependencies are on conda forge except maybe fftw but there is pyfftw on conda forge which we can depend on and that has to ship or build fftw

[mtazzari]

Having galario on conda forge would definitely be an important add-on.

[mtazzari]

I have forked the conda-forge repo and added you as collaborator: https://github.com/mtazzari/staged-recipes

[fredRos]

I found this C++ project that uses cmake and is available on conda http://xtensor.readthedocs.io/en/latest/installation.html I h ave to find out how they do it. The basic script to install seems here https://github.com/conda-forge/xtensor-feedstock/blob/master/recipe/build.sh the rest should be default scaffolding

[fredRos]

In fact FFTW is available on conda, too https://github.com/conda-forge/fftw-feedstock/blob/master/recipe/build.sh

[fredRos]

@mtazzari Please add me as a collaborator for staged-recipes. You mentioned above that you did but I can't push

git … push -v origin master\:refs/heads/master
Pushing to git@github.com:mtazzari/staged-recipes.git
ERROR: Permission to mtazzari/staged-recipes.git denied to fredRos.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[fredRos]

It seems that with our default build options, you can't use cuda on the mac, so conda would ignore it. Perhaps we change that or allow users to set cmake flags with conda install galario

[mtazzari]

I see I did already add you as collaborator but you did not clicked on the activation link. I have now re-added you as collaborator, you should have received an email.

[fredRos]

My bad. Now I accepted :)

[mtazzari]

Re using CUDA on the mac, I think the optimum would be allowing users to set cmake flags with conda install galario, which would add greater flexibility. However, if this is too tricky to get to work, it's fine to have CUDA ignored on a Mac by default: since 2-3 years Macs do not mount NVIDIA cards, not even the Mac pro.

[fredRos]

Ah ok, then I won't worry anymore unless users start pressing us

[fredRos]

At least we should offer openmp support by building with gcc instead of an old clang https://github.com/ilastik/ilastik-build-conda#appendix-compiler-details

Potential problems at runtime, gomp mentioned, too https://github.com/conda-forge/conda-forge.github.io/issues/29

[fredRos]

To test conda build locally

. activate root
conda install conda-build
ls galario/ # build.sh  meta.yaml
conda build --croot /tmp/conda-galario galario/
conda create -n conda-build -c /tmp/conda-galario/ galario nomkl
source activate conda-build
# copy unit tests and make the conda installed lib is imported
cp test_galario.py utils.py /tmp/galario/
py.test test_galario.py 
python -c 'import galario; print(galario.__file__)'

# clean up 
. deactivate 
conda env remove --name conda-build

explanation

What's going on https://conda.io/docs/user-guide/tasks/build-packages/recipe.html

I'm running Debian with an encrypted home folder, which led to

Can't build /home/beaujean/workspace/protoplanetary/staged-recipes/recipes/galario due to environment creation error:
'/home/beaujean/.local/miniconda3/conda-bld/galario_1507119078537/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/conda-meta/history'. Can't write the history file [Errno 36] File name too long: '/home/beaujean/.local/miniconda3/conda-bld/galario_1507119078537/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla'

The suggested fix is to build in a non-encrypted directory https://github.com/conda/conda-build/issues/1331

To install a locally built package with its dependencies https://github.com/conda/conda/issues/1884

[fredRos]

With

requirements:
  build:
    # compilation
    - cmake
    - fftw
    - gcc 4.8.5

in meta.yml, openmp doesn't work

CMake Warning (dev) at /tmp/conda-galario/galario_1507119668514/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/share/cmake-3.9/Modules/FindOpenMP.cmake:200 (if):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "c" will no longer be dereferenced when the policy is
  set to NEW.  Since the policy is not set the OLD behavior will be used.
Call Stack (most recent call first):
  /tmp/conda-galario/galario_1507119668514/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/share/cmake-3.9/Modules/FindOpenMP.cmake:324 (_OPENMP_GET_FLAGS)
  src/CMakeLists.txt:23 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Could NOT find OpenMP_C (missing: OpenMP_pthread_LIBRARY OpenMP_c_LIBRARY) (found version "3.1")
-- Could NOT find OpenMP_CXX (missing: OpenMP_pthread_LIBRARY OpenMP_c_LIBRARY) (found version "3.1")

[fredRos]

conda inspect linkages galario 
galario
-------

fftw-3.3.6-3:
    libfftw3.so.3 (lib/libfftw3.so.3)
    libfftw3f.so.3 (lib/libfftw3f.so.3)

galario-1.0-py36_0:
    libgalario.so (lib/libgalario.so)
    libgalario_single.so (lib/libgalario_single.so)

libgcc-ng-7.2.0-hcbc56d2_1:
    libgcc_s.so.1 (lib/libgcc_s.so.1)

libstdcxx-ng-7.2.0-h24385c6_1:
    libstdc++.so.6 (lib/libstdc++.so.6)

python-3.6.2-0:
    libpython3.6m.so.1.0 (lib/libpython3.6m.so.1.0)

system:
    libc.so.6 (/lib/x86_64-linux-gnu/libc.so.6)
    libdl.so.2 (/lib/x86_64-linux-gnu/libdl.so.2)
    libm.so.6 (/lib/x86_64-linux-gnu/libm.so.6)
    libpthread.so.0 (/lib/x86_64-linux-gnu/libpthread.so.0)
    librt.so.1 (/lib/x86_64-linux-gnu/librt.so.1)
    libutil.so.1 (/lib/x86_64-linux-gnu/libutil.so.1)

not found:

[fredRos]

Useful information on compilers for openmp from https://github.com/conda-forge/fftw-feedstock/issues/20#issuecomment-334318556

Here is an example recipe using OpenMP on both linux and OS X: https://github.com/conda-forge/nfft-feedstock/blob/master/recipe/meta.yaml For both linux and OS X, gcc is listed in the build requirements and libgcc in the run requirements.

I will go with this simple solution for now since it seems to work fine.

So Anaconda (formerly Continuum) has put together some compilers for Linux and macOS that would be a better fit. Would suggest trying those. Here are the docs https://docs.google.com/document/d/103QY204eD82fJzpT7eXy_rV2bKwe6Hx5uAnkMg_xwkI/edit?usp=sharing

This is more advanced and seems to require conda build 3, which seems to not be activated by default for conda forge. Unless we need it, I don't want to go into the extra trouble

[fredRos]

Conda doesn't seem to support cuda dependencies. This package https://odlgroup.github.io/odl/getting_started/installing_extensions.html has instructions on how a user can build the conda package herself. This would be a slight simplification over invoking cmake directly because all dependencies .

But I think it's easier to just state: with conda, there is no cuda support. One has to build galario manually to run on the gpu.

[mtazzari]

From the package you linked I don't understand whether it would imply to have a separate galario-cuda conda package or not.

Of course, it would be great to have the cuda version installed with conda on linux machines. However, I agree that for the time being it is cleaner to state that installing with conda one doesn't get the cuda support.

[fredRos]

galario is available on conda forge https://github.com/conda-forge/galario-feedstock