DDFacet on IDIA nodes without using singularity

IanHeywood commented 4 years ago

Here is a recipe for installing DDFacet inside a virtual environment on IDIA hardware. This method seems to avoid the memory issues that occur when using the singularity container.

The slurm head node does not have virtualenv or cmake installed, so book yourself a worker node on which to perform the installation, e.g.

$ salloc --time=04:00:00 --partition=Main --ntasks=1 --nodes=1 --cpus-per-task=8 --mem=64GB

Create a python3 virtual environment:

$ cd ~/venv
$ virtualenv -p python3 ddfacet

Activate the virtual environment:

$ source ~/venv/ddfacet/bin/activate

Download DDFacet:

$ cd ~/Software
$ git clone https://github.com/saopicc/DDFacet.git

The cmake run needs to have some paths passed to it, so change line 67 of setup.py to:

    subprocess.check_call(["cd %s && cmake -DPYTHON_INCLUDE_DIR=/opt/anaconda3/include/python3.7m -DPYTHON_LIBRARY=/opt/anaconda3/lib %s .. && make" %
                           (path, compile_options if compile_options is not None else ""), ""], shell=True)

Install:

$ pip install numpy
$ pip install -e DDFacet/
$ cd DDFacet
$ python setup.py build

You may have to give it the location of a CASA measures directory. I have no idea where this lives on IDIA so I just pointed it to a location inside a CASA distribution by adding:

measures.directory: /users/ianh/Software/casa-release-5.0.0-218.el7/etc/carta/etc/measures_data

to ~/.casarc.

See if

$ DDF.py -h

produces the required output.

I have tested the submission of venv'd DDFacet jobs via the slurm queue and it seems to work.

o-smirnov commented 4 years ago

Who woulda thunk it possible six months ago!

@bennahugo take a look -- do you think there's a way to get setup.py to pick up the non-standard PYTHON_INCLUDE_DIR and PYTHON_LIBRARY settings, to avoid having to hack it as per the above?

IanHeywood commented 3 years ago

Note that this recipe no longer works as-is with the current master branch of DDFacet. Missing boost:

  Could not find a Python boost library! Please use your package manager to install boost.

ratt-ru / rattatouille

DDFacet on IDIA nodes without using singularity #1