not pip-installable on cgcloud toil-box-templated toil-leader?

[ArtRand]

Hello Toil team,

I'm trying to containerize a pipeline and I've been trying to use toil-lib's docker_call, but it isn't stock installed on my toil cluster (provisioned by cgcloud). Returns:

mesosbox@ip-172-31-33-130:/$ pip install toil-lib
Collecting toil-lib
  Downloading toil-lib-1.1.0.tar.gz
Installing collected packages: toil-lib
  Running setup.py install for toil-lib ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-tptb2F/toil-lib/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-GjmSqk-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/toil_lib
    copying src/toil_lib/urls.py -> build/lib.linux-x86_64-2.7/toil_lib
    copying src/toil_lib/jobs.py -> build/lib.linux-x86_64-2.7/toil_lib
    copying src/toil_lib/programs.py -> build/lib.linux-x86_64-2.7/toil_lib
    copying src/toil_lib/files.py -> build/lib.linux-x86_64-2.7/toil_lib
    copying src/toil_lib/__init__.py -> build/lib.linux-x86_64-2.7/toil_lib
    creating build/lib.linux-x86_64-2.7/toil_lib/test
    copying src/toil_lib/test/test_programs.py -> build/lib.linux-x86_64-2.7/toil_lib/test
    copying src/toil_lib/test/test_lib.py -> build/lib.linux-x86_64-2.7/toil_lib/test
    copying src/toil_lib/test/test_files.py -> build/lib.linux-x86_64-2.7/toil_lib/test
    copying src/toil_lib/test/test_jobs.py -> build/lib.linux-x86_64-2.7/toil_lib/test
    copying src/toil_lib/test/test_urls.py -> build/lib.linux-x86_64-2.7/toil_lib/test
    copying src/toil_lib/test/__init__.py -> build/lib.linux-x86_64-2.7/toil_lib/test
    creating build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/mutation_callers.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/aligners.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/variant_manipulation.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/quantifiers.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/QC.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/spark_tools.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/indexing.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/variant_annotation.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/__init__.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    copying src/toil_lib/tools/preprocessing.py -> build/lib.linux-x86_64-2.7/toil_lib/tools
    running install_lib
    creating /usr/local/lib/python2.7/dist-packages/toil_lib
    error: could not create '/usr/local/lib/python2.7/dist-packages/toil_lib': Permission denied

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-tptb2F/toil-lib/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-GjmSqk-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-tptb2F/toil-lib/

Happy to help fix if you point me in the right direction.

[jvivian]

Hey Art,

You can do the following:

cgcloud ssh -a toil-leader This SSH's you onto the leader as the ubuntu user which has sudo privileges so you can then sudo pip install toil-lib

Instead of doing that though, our group's recommended method for working on the leader / worker is that as the mesosbox user you create a virtualenv with --system-site-packages and then you activate the venv and pip install all your requirements in the venv. Hot deployment works with that model, so that the libraries in the venv will get propagated to all the workers without you having to resort to installing them with cglcoud ssh-cluster.

Please re-open if that doesn't address your issue.