search

cnobles / iGUIDE

Bioinformatic pipeline for identifying dsDNA breaks by marker based incorporation, such as breaks induced by designer nucleases like Cas9.
https://iguide.readthedocs.io/en/latest/
GNU General Public License v3.0
20 stars 9 forks source link

Error in Installing iGUIDE library #72

Open gilfreund opened 4 years ago

gilfreund commented 4 years ago

Hello,

Running the install script fails in install IGUIDE library section.

I am installing on an NFS share, which is also where our shared conda (version 4.8.3) installation and environments exist.

Command: bash install.sh -v --conda /efs/emendo/software/anaconda3 Output:

2020-06-15 06:33:09 [     info] Starting iGUIDE installation...
2020-06-15 06:33:09 [     info]     Conda path:  /efs/emendo/software/anaconda3
2020-06-15 06:33:09 [     info]     iGUIDE src:  /efs/emendo/software/iGUIDE
2020-06-15 06:33:09 [     info]     iGUIDE env:  'iguide'
2020-06-15 06:33:09 [    debug] Components detected:
2020-06-15 06:33:09 [    debug]     Conda:         true
2020-06-15 06:33:23 [    debug]     Environment:   true
2020-06-15 06:33:46 [    debug]     R-package:     true
2020-06-15 06:34:04 [    debug]     Library:       false
2020-06-15 06:34:04 [     info] Conda already installed.
2020-06-15 06:34:04 [     info] Specified environment already exists (use '--update env' to update)
2020-06-15 06:34:04 [     info] iGUIDE Support R-package already installed (use '--update pkg' to update)
2020-06-15 06:34:04 [    debug] Updating $IGUIDE_DIR variable to point to /efs/emendo/software/iGUIDE
2020-06-15 06:34:04 [     info] Setting environmental variables...
2020-06-15 06:34:08 [     info] Installing iGUIDE library...
    ERROR: Command errored out with exit status 1:
     command: /efs/emendo/software/anaconda3/envs/iguide/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/bigdisk/tmp/pip-req-build-wndu64b1/setup.py'"'"'; __file__='"'"'/bigdisk/tmp/pip-req-build-wndu64b1/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /bigdisk/tmp/pip-req-build-wndu64b1/pip-egg-info
         cwd: /bigdisk/tmp/pip-req-build-wndu64b1/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/bigdisk/tmp/pip-req-build-wndu64b1/setup.py", line 19, in <module>
        version = get_iguide_version(),
      File "/bigdisk/tmp/pip-req-build-wndu64b1/setup.py", line 9, in get_iguide_version
        ["git", "rev-parse", "--short", "HEAD"], stdout=PIPE
      File "/efs/emendo/software/anaconda3/envs/iguide/lib/python3.6/subprocess.py", line 403, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/efs/emendo/software/anaconda3/envs/iguide/lib/python3.6/subprocess.py", line 709, in __init__
        restore_signals, start_new_session)
      File "/efs/emendo/software/anaconda3/envs/iguide/lib/python3.6/subprocess.py", line 1344, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'git': 'git'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
2020-06-15 06:34:17 [    error] Error in /efs/emendo/software/iGUIDE/install.sh in function debug_capture on line 53
2020-06-15 06:34:17 [    debug] Processing ./tools/iguidelib
2020-06-15 06:34:35 [    error] Library installation failed!

This might be related to #70 , but the conda environments seems to be be created with no errors. I am attaching a capture of the install process in debug bash install.sh -d --conda /efs/emendo/software/anaconda3 debug.log

Thanks

cnobles commented 4 years ago

I posted this manual install procedure on #70 as well. I should likely update the docs with this process as well. If it's not clear, anything in the code blocks should be input into the command line. My preferred installation has both the miniconda and iGUIDE directories in my home directory.

Installing miniconda

Skip to installing iGUIDE if you already have miniconda or anaconda installed. These can be executed in your home directory.

  1. Get the latest version of miniconda install script.
__conda_url=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
wget -q ${__conda_url} -O miniconda.sh
  1. Installing miniconda through downloaded script. You can choose a different path for the install, here it is installed into the home directory.
__conda_path="~/miniconda3"
bash miniconda.sh -b -p ${__conda_path}
  1. Source conda to activate the installation for use.

source ${__conda_path}/etc/profile.d/conda.sh

Installing iGUIDE

The following commands should be called from within the iGUIDE directory (/path/to/iGUIDE/) after the repository is cloned. (If you need help on this part, just let me know).

  1. Install the conda environment from the requirements file. The name field here can be changed to what you would like to call the environment, default for the install script is 'iguide'.

conda env update --name=iguide --file etc/requirements.yml

  1. After successful creation of the environment, activate the iguide environment (or what you've called it).

conda activate iguide

  1. Install the supporting R-package into the environment.

R CMD INSTALL tools/iguideSupport

  1. Setup your environmental variables.
__iguide_dir=$(pwd)

echo -ne "#/bin/sh\nexport IGUIDE_DIR=${__iguide_dir}" > \
    ${CONDA_PREFIX}/etc/conda/activate.d/env_vars.sh

mkdir -p ${CONDA_PREFIX}/etc/conda/deactivate.d/

echo -ne "#/bin/sh\nunset IGUIDE_DIR" > \
    ${CONDA_PREFIX}/etc/conda/deactivate.d/env_vars.sh
  1. Install the command line interface for iGUIDE using pip.

pip install --upgrade tools/iguidelib/

  1. You should now deactivate and reactivate the environment to initiate the environmental variables.
conda deactivate
conda activate iguide

Testing

Tests to make sure the components were installed correctly.

  1. Test for required R-packages installed.

$(Rscript tools/rscripts/check_for_required_packages.R &> /dev/null) && echo true || echo false

  1. Check to make sure iguideSupport was installed correctly.

$(Rscript tools/rscripts/check_pkgs.R iguideSupport &> /dev/null) && echo true || echo false

  1. Run unit tests for iguideSupport.

`$(Rscript tools/rscripts/check_iguideSupport.R &> /dev/null) && echo true || echo false'

  1. Check to make sure the pip install of the CLI was successful.

command -v iguide &> /dev/null && echo true || echo false

  1. Run tests for iGUIDE, this step will take a little bit to complete if it starts processing the test case. Go grab a coffee, you deserve it getting to this point. :) The test will activate the environment as part of the test, so you should deactivate your environment first and then initiate the test.
conda deactivate
bash etc/tests/test.sh iguide
gilfreund commented 4 years ago

Thanks, but I am still getting the error in step 5 (Install the command line interface for iGUIDE using pip.) It seems similar to the error that happened in the install script

$ pip install --upgrade tools/iguidelib/
Processing ./tools/iguidelib
    ERROR: Command errored out with exit status 1:
     command: /efs/emendo/software/anaconda3/envs/iguide/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/bigdisk/tmp/pip-req-build-s911rl5n/setup.py'"'"'; __file__='"'"'/bigdisk/tmp/pip-req-build-s911rl5n/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /bigdisk/tmp/pip-req-build-s911rl5n/pip-egg-info
         cwd: /bigdisk/tmp/pip-req-build-s911rl5n/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/bigdisk/tmp/pip-req-build-s911rl5n/setup.py", line 19, in <module>
        version = get_iguide_version(),
      File "/bigdisk/tmp/pip-req-build-s911rl5n/setup.py", line 9, in get_iguide_version
        ["git", "rev-parse", "--short", "HEAD"], stdout=PIPE
      File "/efs/emendo/software/anaconda3/envs/iguide/lib/python3.6/subprocess.py", line 403, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/efs/emendo/software/anaconda3/envs/iguide/lib/python3.6/subprocess.py", line 709, in __init__
        restore_signals, start_new_session)
      File "/efs/emendo/software/anaconda3/envs/iguide/lib/python3.6/subprocess.py", line 1344, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'git': 'git'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
cnobles commented 4 years ago

I'll look into it, I haven't seen this error before and can't seem to reproduce it. Looks like it has something to do with CLI interfacing with git, essentially to get the versioning of the software. It may be related to how your system is setup though.

Really sorry about the post earlier, I made a mistake when I was writing the manual install. I had you build the environment from the requirements file, but it's suppose to be built using the build file. This bypasses the environmental solving which can take some time. Try the updated install below and let me know how it goes. My preferred installation has both the miniconda and iGUIDE directories in my home directory.

Installing miniconda

Skip to installing iGUIDE if you already have miniconda or anaconda installed. These can be executed in your home directory.

  1. Get the latest version of miniconda install script.
__conda_url=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
wget -q ${__conda_url} -O miniconda.sh
  1. Installing miniconda through downloaded script. You can choose a different path for the install, here it is installed into the home directory.
__conda_path="~/miniconda3"
bash miniconda.sh -b -p ${__conda_path}
  1. Source conda to activate the installation for use.

source ${__conda_path}/etc/profile.d/conda.sh

Installing iGUIDE

The following commands should be called from within the iGUIDE directory (/path/to/iGUIDE/) after the repository is cloned. (If you need help on this part, just let me know).

  1. Install the conda environment from the requirements file. The name field here can be changed to what you would like to call the environment, default for the install script is 'iguide'. This is the part that changed from above.

conda create --name=iguide --quiet --yes --file etc/build.b1.0.1.txt

  1. After successful creation of the environment, activate the iguide environment (or what you've called it).

conda activate iguide

  1. Install the supporting R-package into the environment.

R CMD INSTALL tools/iguideSupport

  1. Setup your environmental variables.
__iguide_dir=$(pwd)

echo -ne "#/bin/sh\nexport IGUIDE_DIR=${__iguide_dir}" > \
    ${CONDA_PREFIX}/etc/conda/activate.d/env_vars.sh

mkdir -p ${CONDA_PREFIX}/etc/conda/deactivate.d/

echo -ne "#/bin/sh\nunset IGUIDE_DIR" > \
    ${CONDA_PREFIX}/etc/conda/deactivate.d/env_vars.sh
  1. Install the command line interface for iGUIDE using pip.

pip install --upgrade tools/iguidelib/

  1. You should now deactivate and reactivate the environment to initiate the environmental variables.
conda deactivate
conda activate iguide

Testing

Tests to make sure the components were installed correctly.

  1. Test for required R-packages installed.

$(Rscript tools/rscripts/check_for_required_packages.R &> /dev/null) && echo true || echo false

  1. Check to make sure iguideSupport was installed correctly.

$(Rscript tools/rscripts/check_pkgs.R iguideSupport &> /dev/null) && echo true || echo false

  1. Run unit tests for iguideSupport.

`$(Rscript tools/rscripts/check_iguideSupport.R &> /dev/null) && echo true || echo false'

  1. Check to make sure the pip install of the CLI was successful.

command -v iguide &> /dev/null && echo true || echo false

  1. Run tests for iGUIDE, this step will take a little bit to complete if it starts processing the test case. Go grab a coffee, you deserve it getting to this point. :) The test will activate the environment as part of the test, so you should deactivate your environment first and then initiate the test.
conda deactivate
bash etc/tests/test.sh iguide
gilfreund commented 4 years ago

Everithing up to step 5 is fine. I thought it might be git related, so I ran: git rev-parse --short HEAD and got a17daec Which is indeed May 4th commit. I think the problem is elseware, in creating .version in the tmpdirectory.

cnobles commented 4 years ago

Take 3. So the reason for the pip installation of the CLI failing was due to the environmental variables (IGUIDE_DIR) not being initiated. In step 4, we copy these environmental variables into the startup script for the environment. Since we wrote the data into the script, we haven't sourced the script to make the variables present in the environment. Switching steps 5 and 6 will resolve this issue. As least I has in my hands, as I was able to reproduce an error when following the above method at the pip installation. Remeber, my preferred installation has both the miniconda and iGUIDE directories in my home directory. If you have everything still set up, just activate the environment again and try the CLI pip installation. Otherwise, try the below method. If it works out for everyone, then I'll put this last method in the documentation as a manual installation process.

Installing miniconda

Skip to installing iGUIDE if you already have miniconda or anaconda installed. These can be executed in your home directory.

  1. Get the latest version of miniconda install script.
__conda_url=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
wget -q ${__conda_url} -O miniconda.sh
  1. Installing miniconda through downloaded script. You can choose a different path for the install, here it is installed into the home directory.
__conda_path="~/miniconda3"
bash miniconda.sh -b -p ${__conda_path}
  1. Source conda to activate the installation for use.

source ${__conda_path}/etc/profile.d/conda.sh

Installing iGUIDE

The following commands should be called from within the iGUIDE directory (/path/to/iGUIDE/) after the repository is cloned. (If you need help on this part, just let me know).

  1. Install the conda environment from the requirements file. The name field here can be changed to what you would like to call the environment, default for the install script is 'iguide'. This is the part that changed from above.

conda create --name=iguide --quiet --yes --file etc/build.b1.0.1.txt

  1. After successful creation of the environment, activate the iguide environment (or what you've called it).

conda activate iguide

  1. Install the supporting R-package into the environment.

R CMD INSTALL tools/iguideSupport

  1. Setup your environmental variables.
__iguide_dir=$(pwd)

echo -ne "#/bin/sh\nexport IGUIDE_DIR=${__iguide_dir}" > \
    ${CONDA_PREFIX}/etc/conda/activate.d/env_vars.sh

mkdir -p ${CONDA_PREFIX}/etc/conda/deactivate.d/

echo -ne "#/bin/sh\nunset IGUIDE_DIR" > \
    ${CONDA_PREFIX}/etc/conda/deactivate.d/env_vars.sh
  1. You should now deactivate and reactivate the environment to initiate the environmental variables.
conda deactivate
conda activate iguide
  1. Lastely, install the command line interface for iGUIDE using pip.

pip install --upgrade tools/iguidelib/

Testing

Tests to make sure the components were installed correctly.

  1. Test for required R-packages installed.

$(Rscript tools/rscripts/check_for_required_packages.R &> /dev/null) && echo true || echo false

  1. Check to make sure iguideSupport was installed correctly.

$(Rscript tools/rscripts/check_pkgs.R iguideSupport &> /dev/null) && echo true || echo false

  1. Run unit tests for iguideSupport.

`$(Rscript tools/rscripts/check_iguideSupport.R &> /dev/null) && echo true || echo false'

  1. Check to make sure the pip install of the CLI was successful.

command -v iguide &> /dev/null && echo true || echo false

  1. Run tests for iGUIDE, this step will take a little bit to complete if it starts processing the test case. Go grab a coffee, you deserve it getting to this point. :) The test will activate the environment as part of the test, so you should deactivate your environment first and then initiate the test.
conda deactivate
bash etc/tests/test.sh iguide
gilfreund commented 4 years ago

Progress! Switching the steps indeed fixed the errors. Now, in the tests, when I run command -v iguide &> /dev/null && echo true || echo false I get: true However:

conda deactivate
bash etc/tests/test.sh iguide

Fails.

This is due to the line source ${HOME}/miniconda3/etc/profile.d/conda.sh That assumes that miniconda is in the home directory, and that this is using miniconda (I am using anaconda). After changing this to our conda location, this works.

gilfreund commented 4 years ago

It might make more sense to require the the tests to be run after the conda environment is active (will remove some of the guesswork / assumptions from the test script) or add a parameters to the test script pointing to the conda environment

cnobles commented 4 years ago

Great point, when I wrote this portion of the install script, I wasn't considering others may be using anaconda. The test bash script does try to activate the conda environment (as you've pointed out) prior to initiating the tests, but this assumes miniconda is installed. I'll keep this issue open till I make the update to the install and test scripts. Thanks for this suggestion!