Installation issues (and how to resolve)

[binnietom]

Hello, The conda environment '21CMMC' that is made during py21cmmc isntallation currently doesnt work for me - I've had a meal of a time trying to run it on imperial's HPC, but here is a solution:

The following instructions produces a working environment.

##make sure the old stuff has been cleaned
$ module load anaconda3/personal
$ conda remove -n 21CMMC --all -y  
$ module purge

##Create the environment
$ module load anaconda3/personal
$conda create -n 21cmmc python=3.7 gsl fftw
$source activate 21cmmc
$conda install numpy scipy click pyyaml cffi astropy h5py
$conda update pytest

##installation (+library linkage)
$cd $EPHEMERAL
$git clone https://github.com/21cmfast/21CMMC.git
$cd ./21CMMC
$INCLUDE=$CONDA_PREFIX/include DEBUG=True GSL_LIB=$CONDA_PREFIX/lib $GSL_INC=$CONDA_PREFIX/include FFTW_INC=$CONDA_PREFIX/include 
$pip install -e .
$pip install -r requirements_dev.txt

##test with 
$21cmfast --help

p.s. if this produces gsl library linking errors, it is because the pip cache is linked to the wrong place.

[steven-murray]

Thanks @binnietom. I'm going to move this to 21cmFAST, which is really what this issue is about.

[steven-murray]

@binnietom, I would like to reduce this to a minimal working example, and hopefully include it in the docs to aid future users. Obviously, the module load stuff is very much HPC-specific (and specific to your HPC). I would appreciate very much if you could write those details down somewhere for future users at your institution. But let's try to capture the essence of why the current default instructions didn't work for you, and what you had to do to fix the problem.

Modulo the module load stuff, does this work for you?:

# Create the environment
$ conda create -n 21cmfast python=3.7 gsl fftw
$ conda activate 21cmfast
$ conda install numpy scipy click pyyaml cffi astropy h5py

# installation (+library linkage)
$ git clone https://github.com/21cmfast/21cmfast.git
$ cd 21cmFAST
$ INCLUDE=$CONDA_PREFIX/include LIB=$CONDA_PREFIX/lib  pip install .

# test: 
$ 21cmfast --help

[binnietom]

Hi Steven,
Apologies, I didn't realise that was over specific - I'm just trying to be as transparent as a can with this! Thanks for putting it in the appropriate place. The module load anaconda3/personal is equivalent to installing anaconda with python 3.

[binnietom]

It installs on my desktop ( matplotlib is also needed in that conda install list ) .

But I need to install powerbox - do you have instructions on how to do this with conda before I try with pip?

(21cmfast) tab216@dyn202-194:~/Desktop/21cmmc_MODULES/21cmfast$ 21cmfast --helpTraceback (most recent call last):
  File "/home/tab216/miniconda3/envs/21cmfast/bin/21cmfast", line 5, in <module>
    from py21cmfast.cli import main
  File "/home/tab216/miniconda3/envs/21cmfast/lib/python3.7/site-packages/py21cmfast/cli.py", line 14, in <module>
    import powerbox
ModuleNotFoundError: No module named 'powerbox'

However It won't install on Jonathan's workstation (output below):

(21cmfast) tab216@ph-jpritcha-1:~/21cmfast$ INCLUDE=$CONDA_PREFIX/include LIB=$CONDA_PREFIX/lib pip install --user .
Processing /home/tab216/21cmfast
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /home/tab216/.local/lib/python3.5/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6regbffv/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (41 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/__main__.py", line 19, in <module>
      sys.exit(_main())
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 73, in main
      command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/commands/__init__.py", line 96, in create_command
      module = importlib.import_module(module_path)
    File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 986, in _gcd_import
    File "<frozen importlib._bootstrap>", line 969, in _find_and_load
    File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 665, in exec_module
    File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/commands/install.py", line 24, in <module>
      from pip._internal.cli.req_command import RequirementCommand
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
      from pip._internal.req.constructors import (
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/req/__init__.py", line 11, in <module>
      from .req_file import parse_requirements
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/req/req_file.py", line 25, in <module>
      from pip._internal.req.constructors import (
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/req/constructors.py", line 28, in <module>
      from pip._internal.req.req_install import InstallRequirement
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 30, in <module>
      from pip._internal.operations.install.wheel import install_wheel
    File "/home/tab216/.local/lib/python3.5/site-packages/pip/_internal/operations/install/wheel.py", line 10, in <module>
      import compileall
    File "/usr/lib/python3.5/compileall.py", line 20, in <module>
      from concurrent.futures import ProcessPoolExecutor
    File "/home/tab216/.local/lib/python3.5/site-packages/concurrent/futures/__init__.py", line 8, in <module>
      from concurrent.futures._base import (FIRST_COMPLETED,
    File "/home/tab216/.local/lib/python3.5/site-packages/concurrent/futures/_base.py", line 381
      raise exception_type, self._exception, self._traceback
                          ^
  SyntaxError: invalid syntax
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /home/tab216/.local/lib/python3.5/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6regbffv/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

[binnietom]

...powerbox installed easily with the envs/21cmfast/bin/pip and 21cmfast --help now runs on my desktop.

Although I'm not sure where to begin with the workstation error - I guess a version discrepancy somewhere?

[steven-murray]

I apologise for needing powerbox -- I will fix that soon (it's only needed for a single function in the CLI, so it shouldn't be required globally). But it should be installed directly with pip, as you found.

As for the workstation -- can you see what echo $CONDA_PREFIX is? And also what which pip outputs?

[binnietom]

I get: (21cmfast) tab216@ph-jpritcha-1:~$ echo $CONDA_PREFIX /home/tab216/anaconda3/envs/21cmfast

On my desktop (where it installed ok): (21cmfast) tab216@dyn202-194:~$ which pip /home/tab216/miniconda3/envs/21cmfast/bin/pip

On the workstation it looks to not be linking with the environment. (21cmfast) tab216@ph-jpritcha-1:~$ which pip /home/tab216/.local/bin/pip

I just tried: (21cmfast) tab216@ph-jpritcha-1:~/21cmfast$ pip uninstall pip

$ which pip /home/tab216/anaconda3/envs/21cmfast/bin/pip

$ INCLUDE=$CONDA_PREFIX/include LIB=$CONDA_PREFIX/lib pip install . -bash: /home/tab216/.local/bin/pip: No such file or directory

$ conda install pip

and now it works (after installing matplotlib and powerbox) :)

Thanks for your help.

[steven-murray]

Awesome! So it would seem that you had a weird python/pip installation which was resolved by uninstalling pip and re-installing with conda.

But the installation instructions for 21cmFAST are correct if those things are right?

[binnietom]

• to be pernickety the there was one type-o with case so its:

Create the environment

$ conda create -n 21cmfast python=3.7 gsl fftw
$ conda activate 21cmfast
$ conda install numpy scipy click pyyaml cffi astropy h5py matplotlib
$ pip install powerbox

installation (+library linkage)

$ git clone https://github.com/21cmfast/21cmfast.git
$ cd 21cmfast
$ INCLUDE=$CONDA_PREFIX/include LIB=$CONDA_PREFIX/lib  pip install .

test:

$ 21cmfast --help

Otherwise yes. Thanks, Tom

[steven-murray]

Excellent. Thanks -- this will be a good future reference for people with installation issues.

I think one useful thing I've learned from this is that gsl and fftw can be installed via conda. This is not currently in the documentation. I will add it.