Utilities not in $PATH - Githubissues

maojn commented 3 years ago

Hi,

The /usr/local/bin/HiC-Pro/bin/utils is not in the $PATH of the singularity container. Is it possible to include it and rebuilt the container?

Regards,

Jean

nservant commented 3 years ago

Hi, Good comment ! I did it in the last 3.1.0 version. Did you try it ? https://github.com/nservant/HiC-Pro/blob/master/Singularity Best

maojn commented 3 years ago

Actually I downloaded the singularity image file and found that util directory is not in the $PATH. However, I also tried to build the singularity container from the definition file you pointed to but failed with the following error. It will be highly appreciated if you could build the updated singularity container for downloading:

...
...
+ export PATH=/usr/local/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ conda update conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /usr/local/conda

  added / updated specs:
    - conda  

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    charset-normalizer-2.0.4   |     pyhd3eb1b0_0          35 KB
    idna-3.2                   |     pyhd3eb1b0_0          48 KB
    openssl-1.1.1l             |       h7f8727e_0         2.5 MB
    requests-2.26.0            |     pyhd3eb1b0_0          59 KB
    setuptools-58.0.4          |   py39h06a4308_0         790 KB
    tqdm-4.62.2                |     pyhd3eb1b0_1          96 KB
    tzdata-2021a               |       h5d7bf9c_0         111 KB
    ------------------------------------------------------------
                                           Total:         3.6 MB

The following NEW packages will be INSTALLED:

  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0

The following packages will be REMOVED:

  chardet-4.0.0-py39h06a4308_1003
  pip-21.1.3-py39h06a4308_0
  wheel-0.36.2-pyhd3eb1b0_0

The following packages will be UPDATED:

  idna                                    2.10-pyhd3eb1b0_0 --> 3.2-pyhd3eb1b0_0
  openssl                                 1.1.1k-h27cfd23_0 --> 1.1.1l-h7f8727e_0
  requests                              2.25.1-pyhd3eb1b0_0 --> 2.26.0-pyhd3eb1b0_0
  setuptools                          52.0.0-py39h06a4308_0 --> 58.0.4-py39h06a4308_0
  tqdm                                  4.61.2-pyhd3eb1b0_1 --> 4.62.2-pyhd3eb1b0_1

The following packages will be DOWNGRADED:

  tzdata                                   2021a-h52ac0ba_0 --> 2021a-h5d7bf9c_0

Proceed ([y]/n)?
Downloading and Extracting Packages
tzdata-2021a         | 111 KB    | ################################################################################### | 100%
tqdm-4.62.2          | 96 KB     | ################################################################################### | 100%
openssl-1.1.1l       | 2.5 MB    | ################################################################################### | 100%
idna-3.2             | 48 KB     | ################################################################################### | 100%
charset-normalizer-2 | 35 KB     | ################################################################################### | 100%
requests-2.26.0      | 59 KB     | ################################################################################### | 100%
setuptools-58.0.4    | 790 KB    | ################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
+ conda clean --packages -y
WARNING: /root/.conda/pkgs does not exist
Cache location:
There are no unused packages to remove
+ echo #! /bin/bash\n\n
+ conda init bash
no change     /usr/local/conda/condabin/conda
no change     /usr/local/conda/bin/conda
no change     /usr/local/conda/bin/conda-env
no change     /usr/local/conda/bin/activate
no change     /usr/local/conda/bin/deactivate
no change     /usr/local/conda/etc/profile.d/conda.sh
no change     /usr/local/conda/etc/fish/conf.d/conda.fish
no change     /usr/local/conda/shell/condabin/Conda.psm1
no change     /usr/local/conda/shell/condabin/conda-hook.ps1
no change     /usr/local/conda/lib/python3.9/site-packages/xontrib/conda.xsh
no change     /usr/local/conda/etc/profile.d/conda.csh
modified      /root/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

+ echo Installing conda environment ...
Installing conda environment ...
+ conda env create -f /opt/environment.yml -n hicpro

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
    Traceback (most recent call last):
      File "/usr/local/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/usr/local/conda/lib/python3.9/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/usr/local/conda/lib/python3.9/site-packages/conda_env/cli/main_create.py", line 88, in execute
        spec = specs.detect(name=name, filename=get_filename(args.file), directory=os.getcwd())
      File "/usr/local/conda/lib/python3.9/site-packages/conda_env/specs/__init__.py", line 43, in detect
        if spec.can_handle():
      File "/usr/local/conda/lib/python3.9/site-packages/conda_env/specs/yaml_file.py", line 18, in can_handle
        self._environment = env.from_file(self.filename)
      File "/usr/local/conda/lib/python3.9/site-packages/conda_env/env.py", line 166, in from_file
        return from_yaml(yamlstr, filename=filename)
      File "/usr/local/conda/lib/python3.9/site-packages/conda_env/env.py", line 143, in from_yaml
        data = yaml_safe_load(yamlstr)
      File "/usr/local/conda/lib/python3.9/site-packages/conda/common/serialize.py", line 67, in yaml_safe_load
        return yaml.safe_load(string, version="1.2")
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/main.py", line 980, in safe_load
        return load(stream, SafeLoader, version)
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/main.py", line 935, in load
        return loader._constructor.get_single_data()
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/constructor.py", line 109, in get_single_data
        node = self.composer.get_single_node()
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/composer.py", line 78, in get_single_node
        document = self.compose_document()
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/composer.py", line 101, in compose_document
        node = self.compose_node(None, None)
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/composer.py", line 138, in compose_node
        node = self.compose_mapping_node(anchor)
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/composer.py", line 211, in compose_mapping_node
        while not self.parser.check_event(MappingEndEvent):
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/parser.py", line 141, in check_event
        self.current_event = self.state()
      File "/usr/local/conda/lib/python3.9/site-packages/ruamel_yaml/parser.py", line 577, in parse_block_mapping_key
        raise ParserError(
    ruamel_yaml.parser.ParserError: while parsing a block mapping
      in "<unicode string>", line 2, column 1:
        name: HiC-Pro_v3.1.0
        ^ (line: 2)
    expected <block end>, but found '-'
      in "<unicode string>", line 24, column 1:
        - bioconda::multiqc=1.11=pyhdfd7 ...
        ^ (line: 24)

`$ /usr/local/conda/bin/conda-env create -f /opt/environment.yml -n hicpro`

  environment variables:
                 CIO_TEST=<not set>
  CONDA_AUTO_UPDATE_CONDA=false
               CONDA_ROOT=/usr/local/conda
           CURL_CA_BUNDLE=<not set>
          LD_LIBRARY_PATH=/.singularity.d/libs
                     PATH=/usr/local/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
                          :/sbin:/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : None
       user config file : /root/.condarc
populated config files :
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __linux=5.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /usr/local/conda  (writable)
      conda av data dir : /usr/local/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /usr/local/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /usr/local/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.3 requests/2.26.0 CPython/3.9.5 Linux/5.4.0-1052-gcp ubuntu/20.04.3 glibc/2.31
                UID:GID : 0:0
             netrc file : None
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

FATAL:   While performing build: while running engine: exit status 1

nservant commented 3 years ago

Could you try with this image ; https://zerkalo.curie.fr/partage/HiC-Pro/singularity_images/hicpro_3.1.0_ubuntu.img

maojn commented 3 years ago

Thank you for creating the image for downloading. There are two minor issues:

Could you add /usr/local/bin/HiC-Pro_3.1.0/scripts/ to the $PATH as well? This way these scripts can be accessed in the container.

When I tried the container, I saw some output at the beginning. Not sure if it matters.


$ singularity exec hicpro_3.1.0_ubuntu.img HiC-Pro -h
source: /usr/local/conda/envs/hicpro/etc/conda/activate.d/activate-binutils_linux-64.sh:8:18: parameter expansion requires a l
iteral
source: /usr/local/conda/envs/hicpro/etc/conda/activate.d/activate-gcc_linux-64.sh:8:18: parameter expansion requires a litera
l
source: /usr/local/conda/envs/hicpro/etc/conda/activate.d/activate-gfortran_linux-64.sh:8:18: parameter expansion requires a l
iteral
source: /usr/local/conda/envs/hicpro/etc/conda/activate.d/activate-gxx_linux-64.sh:8:18: parameter expansion requires a litera
l
usage : HiC-Pro -i INPUT -o OUTPUT -c CONFIG [-s ANALYSIS_STEP] [-p] [-h] [-v]
Use option -h|--help for more information

HiC-Pro 3.1.0

OPTIONS ... ...



Thank you for your help.

nservant / HiC-Pro

Utilities not in $PATH #483

HiC-Pro 3.1.0