Closed abueg closed 2 years ago
pichuan
commented
2 years ago In this file: https://github.com/google/deepconsensus/blob/r0.2/requirements.txt
Pin pysam:
pysam==0.19.0See if that helps with the exception.
The index "error" : It shouldn't affect the run. You can ignore it.
Let us know if that works or not.
abueg
commented
2 years ago Hi @pichuan, thank you for your response! I added the pysam pin to requirements.txt and re-installed into a new venv and tried run_all_tests.sh again, but it is giving the same exception.
install_edit.sh:
(deepconsensus_venv_6) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ cat install_edit.sh
#!/bin/bash
# Copyright (c) 2021, Google Inc.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification,
# are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of Google Inc. nor the names of its contributors
# may be used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# Usage: source install.sh
#
# This script installs all the packages required to build DeepConsensus.
#
# This script will run as-is on Ubuntu 20.04.
#
# We also assume that apt-get is already installed and available.
function note_build_stage {
echo "========== [$(date)] Stage '${1}' starting"
}
# Update package list
################################################################################
# Install pip
################################################################################
python3 -m pip install --upgrade pip
# Update PATH so that newly installed pip is the one we actually use.
export PATH="$SCRATCH/venvs/deepconsensus_venv_6/bin:$PATH"
echo "$(pip --version)"
# Install python packages used by DeepConsensus.
################################################################################
python3 -m pip install -r requirements.txt
python3 -m pip install "intel-tensorflow>=2.4.0,<=2.7.0"and here is the output from installation:
(deepconsensus_venv_6) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ export PATH="$SCRATCH/venvs/deepconsensus_venv_6/bin:$PATH"
(deepconsensus_venv_6) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ source install_edit.sh
Collecting pip
Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
Successfully uninstalled pip-20.2.3
Successfully installed pip-22.1.2
pip 22.1.2 from /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages/pip (python 3.8)
Collecting numpy>=1.19
Using cached numpy-1.23.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Collecting pandas>=1.1
Using cached pandas-1.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
Collecting tf-models-official<=2.7.0,>=2.4.0
Using cached tf_models_official-2.7.0-py2.py3-none-any.whl (1.8 MB)
Collecting ml_collections>=0.1.0
Using cached ml_collections-0.1.1.tar.gz (77 kB)
Preparing metadata (setup.py) ... done
Collecting absl-py>=0.13.0
Using cached absl_py-1.1.0-py3-none-any.whl (123 kB)
Collecting pysam==0.19.0
Using cached pysam-0.19.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (16.3 MB)
Collecting python-dateutil>=2.8.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1
Using cached pytz-2022.1-py2.py3-none-any.whl (503 kB)
Collecting psutil>=5.4.3
Using cached psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (284 kB)
Collecting pycocotools
Using cached pycocotools-2.0.4-cp38-cp38-linux_x86_64.whl
Collecting pyyaml>=5.1
Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Collecting seqeval
Using cached seqeval-1.2.2.tar.gz (43 kB)
Preparing metadata (setup.py) ... done
Collecting Cython
Using cached Cython-0.29.30-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
Collecting gin-config
Using cached gin_config-0.5.0-py3-none-any.whl (61 kB)
Collecting tf-slim>=1.1.0
Using cached tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)
Collecting oauth2client
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting tensorflow>=2.7.0
Using cached tensorflow-2.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
Collecting py-cpuinfo>=3.3.0
Using cached py-cpuinfo-8.0.0.tar.gz (99 kB)
Preparing metadata (setup.py) ... done
Collecting tensorflow-addons
Using cached tensorflow_addons-0.17.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting matplotlib
Using cached matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
Collecting tensorflow-model-optimization>=0.4.1
Using cached tensorflow_model_optimization-0.7.2-py2.py3-none-any.whl (237 kB)
Collecting tensorflow-hub>=0.6.0
Using cached tensorflow_hub-0.12.0-py2.py3-none-any.whl (108 kB)
Collecting google-api-python-client>=1.6.7
Downloading google_api_python_client-2.52.0-py2.py3-none-any.whl (8.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.7/8.7 MB 82.6 MB/s eta 0:00:00
Collecting six
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting kaggle>=1.3.9
Using cached kaggle-1.5.12.tar.gz (58 kB)
Preparing metadata (setup.py) ... done
Collecting opencv-python-headless
Using cached opencv_python_headless-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (48.3 MB)
Collecting tensorflow-datasets
Using cached tensorflow_datasets-4.6.0-py3-none-any.whl (4.3 MB)
Collecting Pillow
Using cached Pillow-9.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting scipy>=0.19.1
Using cached scipy-1.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.6 MB)
Collecting sentencepiece
Using cached sentencepiece-0.1.96-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting sacrebleu
Using cached sacrebleu-2.1.0-py3-none-any.whl (92 kB)
Collecting tensorflow-text>=2.7.0
Using cached tensorflow_text-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
Collecting contextlib2
Using cached contextlib2-21.6.0-py2.py3-none-any.whl (13 kB)
Collecting google-auth-httplib2>=0.1.0
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<5,>=3.0.1
Using cached uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.8.2-py3-none-any.whl (114 kB)
Collecting httplib2<1dev,>=0.15.0
Using cached httplib2-0.20.4-py3-none-any.whl (96 kB)
Collecting google-auth<3.0.0dev,>=1.19.0
Using cached google_auth-2.8.0-py2.py3-none-any.whl (164 kB)
Collecting certifi
Using cached certifi-2022.6.15-py3-none-any.whl (160 kB)
Collecting requests
Using cached requests-2.28.0-py3-none-any.whl (62 kB)
Collecting tqdm
Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting python-slugify
Using cached python_slugify-6.1.2-py2.py3-none-any.whl (9.4 kB)
Collecting urllib3
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting tensorflow-io-gcs-filesystem>=0.23.1
Using cached tensorflow_io_gcs_filesystem-0.26.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)
Collecting wrapt>=1.11.0
Using cached wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (81 kB)
Collecting astunparse>=1.6.0
Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting termcolor>=1.1.0
Using cached termcolor-1.1.0.tar.gz (3.9 kB)
Preparing metadata (setup.py) ... done
Collecting tensorboard<2.10,>=2.9
Using cached tensorboard-2.9.1-py3-none-any.whl (5.8 MB)
Collecting keras<2.10.0,>=2.9.0rc0
Using cached keras-2.9.0-py2.py3-none-any.whl (1.6 MB)
Collecting opt-einsum>=2.3.2
Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting tensorflow-estimator<2.10.0,>=2.9.0rc0
Using cached tensorflow_estimator-2.9.0-py2.py3-none-any.whl (438 kB)
Collecting typing-extensions>=3.6.6
Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting gast<=0.4.0,>=0.2.1
Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting keras-preprocessing>=1.1.1
Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting h5py>=2.9.0
Using cached h5py-3.7.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
Collecting protobuf<3.20,>=3.9.2
Using cached protobuf-3.19.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting grpcio<2.0,>=1.24.3
Using cached grpcio-1.47.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
Collecting libclang>=13.0.0
Using cached libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)
Requirement already satisfied: setuptools in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorflow>=2.7.0->tf-models-official<=2.7.0,>=2.4.0->-r requirements.txt (line 3)) (49.2.1)
Collecting flatbuffers<2,>=1.12
Using cached flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
Collecting google-pasta>=0.1.1
Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting dm-tree~=0.1.1
Using cached dm_tree-0.1.7-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (142 kB)
Collecting fonttools>=4.22.0
Using cached fonttools-4.33.3-py3-none-any.whl (930 kB)
Collecting pyparsing>=2.2.1
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting cycler>=0.10
Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
Using cached kiwisolver-1.4.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting tabulate>=0.8.9
Using cached tabulate-0.8.10-py3-none-any.whl (29 kB)
Collecting regex
Using cached regex-2022.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (764 kB)
Collecting portalocker
Using cached portalocker-2.4.0-py2.py3-none-any.whl (16 kB)
Collecting colorama
Using cached colorama-0.4.5-py2.py3-none-any.whl (16 kB)
Collecting scikit-learn>=0.21.3
Using cached scikit_learn-1.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31.2 MB)
Collecting typeguard>=2.7
Using cached typeguard-2.13.3-py3-none-any.whl (17 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting etils[epath]
Using cached etils-0.6.0-py3-none-any.whl (98 kB)
Collecting tensorflow-metadata
Using cached tensorflow_metadata-1.9.0-py3-none-any.whl (51 kB)
Collecting importlib-resources
Using cached importlib_resources-5.8.0-py3-none-any.whl (28 kB)
Collecting dill
Using cached dill-0.3.5.1-py2.py3-none-any.whl (95 kB)
Collecting promise
Using cached promise-2.3.tar.gz (19 kB)
Preparing metadata (setup.py) ... done
Collecting wheel<1.0,>=0.23.0
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting googleapis-common-protos<2.0dev,>=1.56.2
Using cached googleapis_common_protos-1.56.3-py2.py3-none-any.whl (211 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting joblib>=1.0.0
Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting markdown>=2.6.8
Using cached Markdown-3.3.7-py3-none-any.whl (97 kB)
Collecting werkzeug>=1.0.1
Using cached Werkzeug-2.1.2-py3-none-any.whl (224 kB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
Collecting zipp
Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting text-unidecode>=1.3
Using cached text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
Using cached importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Using legacy 'setup.py install' for ml_collections, since package 'wheel' is not installed.
Using legacy 'setup.py install' for kaggle, since package 'wheel' is not installed.
Using legacy 'setup.py install' for py-cpuinfo, since package 'wheel' is not installed.
Using legacy 'setup.py install' for seqeval, since package 'wheel' is not installed.
Using legacy 'setup.py install' for termcolor, since package 'wheel' is not installed.
Using legacy 'setup.py install' for promise, since package 'wheel' is not installed.
Installing collected packages: text-unidecode, termcolor, tensorboard-plugin-wit, sentencepiece, pytz, pysam, pyasn1, py-cpuinfo, libclang, keras, gin-config, flatbuffers, dm-tree, zipp, wrapt, wheel, werkzeug, urllib3, uritemplate, typing-extensions, typeguard, tqdm, toml, threadpoolctl, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, tabulate, six, rsa, regex, pyyaml, python-slugify, pyparsing, pyasn1-modules, psutil, protobuf, portalocker, Pillow, oauthlib, numpy, kiwisolver, joblib, idna, gast, fonttools, etils, dill, Cython, cycler, contextlib2, colorama, charset-normalizer, certifi, cachetools, absl-py, tf-slim, tensorflow-model-optimization, tensorflow-hub, scipy, sacrebleu, requests, python-dateutil, promise, packaging, opt-einsum, opencv-python-headless, ml_collections, keras-preprocessing, importlib-resources, importlib-metadata, httplib2, h5py, grpcio, googleapis-common-protos, google-pasta, google-auth, astunparse, tensorflow-metadata, tensorflow-addons, scikit-learn, requests-oauthlib, pandas, oauth2client, matplotlib, markdown, kaggle, google-auth-httplib2, google-api-core, tensorflow-datasets, seqeval, pycocotools, google-auth-oauthlib, google-api-python-client, tensorboard, tensorflow, tensorflow-text, tf-models-official
Running setup.py install for termcolor ... done
Running setup.py install for py-cpuinfo ... done
Running setup.py install for promise ... done
Running setup.py install for ml_collections ... done
Running setup.py install for kaggle ... done
Running setup.py install for seqeval ... done
Successfully installed Cython-0.29.30 Pillow-9.1.1 absl-py-1.1.0 astunparse-1.6.3 cachetools-5.2.0 certifi-2022.6.15 charset-normalizer-2.0.12 colorama-0.4.5 contextlib2-21.6.0 cycler-0.11.0 dill-0.3.5.1 dm-tree-0.1.7 etils-0.6.0 flatbuffers-1.12 fonttools-4.33.3 gast-0.4.0 gin-config-0.5.0 google-api-core-2.8.2 google-api-python-client-2.52.0 google-auth-2.8.0 google-auth-httplib2-0.1.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 googleapis-common-protos-1.56.3 grpcio-1.47.0 h5py-3.7.0 httplib2-0.20.4 idna-3.3 importlib-metadata-4.12.0 importlib-resources-5.8.0 joblib-1.1.0 kaggle-1.5.12 keras-2.9.0 keras-preprocessing-1.1.2 kiwisolver-1.4.3 libclang-14.0.1 markdown-3.3.7 matplotlib-3.5.2 ml_collections-0.1.1 numpy-1.23.0 oauth2client-4.1.3 oauthlib-3.2.0 opencv-python-headless-4.6.0.66 opt-einsum-3.3.0 packaging-21.3 pandas-1.4.3 portalocker-2.4.0 promise-2.3 protobuf-3.19.4 psutil-5.9.1 py-cpuinfo-8.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycocotools-2.0.4 pyparsing-3.0.9 pysam-0.19.0 python-dateutil-2.8.2 python-slugify-6.1.2 pytz-2022.1 pyyaml-6.0 regex-2022.6.2 requests-2.28.0 requests-oauthlib-1.3.1 rsa-4.8 sacrebleu-2.1.0 scikit-learn-1.1.1 scipy-1.8.1 sentencepiece-0.1.96 seqeval-1.2.2 six-1.16.0 tabulate-0.8.10 tensorboard-2.9.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.9.1 tensorflow-addons-0.17.1 tensorflow-datasets-4.6.0 tensorflow-estimator-2.9.0 tensorflow-hub-0.12.0 tensorflow-io-gcs-filesystem-0.26.0 tensorflow-metadata-1.9.0 tensorflow-model-optimization-0.7.2 tensorflow-text-2.9.0 termcolor-1.1.0 text-unidecode-1.3 tf-models-official-2.7.0 tf-slim-1.1.0 threadpoolctl-3.1.0 toml-0.10.2 tqdm-4.64.0 typeguard-2.13.3 typing-extensions-4.2.0 uritemplate-4.1.1 urllib3-1.26.9 werkzeug-2.1.2 wheel-0.37.1 wrapt-1.14.1 zipp-3.8.0
Collecting intel-tensorflow<=2.7.0,>=2.4.0
Using cached intel_tensorflow-2.7.0-cp38-cp38-manylinux2010_x86_64.whl (186.4 MB)
Requirement already satisfied: tensorboard~=2.6 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (2.9.1)
Requirement already satisfied: numpy>=1.14.5 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.23.0)
Requirement already satisfied: gast<0.5.0,>=0.2.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (0.4.0)
Requirement already satisfied: absl-py>=0.4.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.1.0)
Requirement already satisfied: astunparse>=1.6.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.6.3)
Requirement already satisfied: protobuf>=3.9.2 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (3.19.4)
Requirement already satisfied: six>=1.12.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.16.0)
Requirement already satisfied: google-pasta>=0.1.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (0.2.0)
Collecting keras<2.8,>=2.7.0rc0
Using cached keras-2.7.0-py2.py3-none-any.whl (1.3 MB)
Requirement already satisfied: libclang>=9.0.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (14.0.1)
Requirement already satisfied: wheel<1.0,>=0.32.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (0.37.1)
Requirement already satisfied: wrapt>=1.11.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.14.1)
Requirement already satisfied: h5py>=2.9.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (3.7.0)
Requirement already satisfied: flatbuffers<3.0,>=1.12 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.12)
Requirement already satisfied: keras-preprocessing>=1.1.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.1.2)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.47.0)
Collecting tensorflow-estimator<2.8,~=2.7.0rc0
Using cached tensorflow_estimator-2.7.0-py2.py3-none-any.whl (463 kB)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.21.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (0.26.0)
Requirement already satisfied: termcolor>=1.1.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (1.1.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (3.3.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from intel-tensorflow<=2.7.0,>=2.4.0) (4.2.0)
Requirement already satisfied: requests<3,>=2.21.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (2.28.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (0.4.6)
Requirement already satisfied: google-auth<3,>=1.6.3 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (2.8.0)
Requirement already satisfied: markdown>=2.6.8 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (3.3.7)
Requirement already satisfied: werkzeug>=1.0.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (2.1.2)
Requirement already satisfied: setuptools>=41.0.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (49.2.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (1.8.1)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (0.6.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (5.2.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (4.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (4.12.0)
Requirement already satisfied: idna<4,>=2.5 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (2.0.12)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (2022.6.15)
Requirement already satisfied: zipp>=0.5 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (3.8.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_6/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0) (3.2.0)
Installing collected packages: tensorflow-estimator, keras, intel-tensorflow
Attempting uninstall: tensorflow-estimator
Found existing installation: tensorflow-estimator 2.9.0
Uninstalling tensorflow-estimator-2.9.0:
Successfully uninstalled tensorflow-estimator-2.9.0
Attempting uninstall: keras
Found existing installation: keras 2.9.0
Uninstalling keras-2.9.0:
Successfully uninstalled keras-2.9.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.9.1 requires keras<2.10.0,>=2.9.0rc0, but you have keras 2.7.0 which is incompatible.
tensorflow 2.9.1 requires tensorflow-estimator<2.10.0,>=2.9.0rc0, but you have tensorflow-estimator 2.7.0 which is incompatible.
Successfully installed intel-tensorflow-2.7.0 keras-2.7.0 tensorflow-estimator-2.7.0making sure the version of pysam in this venv:
(deepconsensus_venv_6) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ cat $SCRATCH/venvs/deepconsensus_venv_6/lib/python3.8/site-packages/pysam/version.py
# pysam versioning information
__version__ = "0.19.0"
__samtools_version__ = "1.15"
__bcftools_version__ = "1.15"
__htslib_version__ = "1.15"I ran run_all_tests.sh again on a node using slurm, and the full output is in this gist: https://gist.github.com/abueg/0362d447ce32556f93f0dfa5a6e070f2
And here is the exception, seems the same as before:
OK
Exception ignored in: <function Pool.__del__ at 0x7ff8c1cd9430>
Traceback (most recent call last):
File "/vggpfs/fs3/vgl/store/labueg/anaconda3/lib/python3.8/multiprocessing/pool.py", line 268, in __del__
self._change_notifier.put(None)
File "/vggpfs/fs3/vgl/store/labueg/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/vggpfs/fs3/vgl/store/labueg/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/vggpfs/fs3/vgl/store/labueg/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/vggpfs/fs3/vgl/store/labueg/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptorHi @abueg ,
Usually I tested with pip install .[cpu] and I usually didn't use venv.
I tested with venv, and used pip install .[cpu], and I'm getting a different error:
Building wheels for collected packages: pycocotools
Building wheel for pycocotools (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for pycocotools (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-38/pycocotools
running build_ext
skipping 'pycocotools/_mask.c' Cython extension (up-to-date)
building 'pycocotools._mask' extension
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/common
creating build/temp.linux-x86_64-cpython-38/pycocotools
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/pip-build-env-_q4_ep0v/overlay/lib/python3.8/site-packages/numpy/core/include -I./common -I/home/pichuan/deepconsensus_venv_1/include -I/usr/include/python3.8 -c ./common/maskApi.c -o build/temp.linux-x86_64-cpython-38/./common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycocotools
Failed to build pycocotools
ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projectsI'll have to poke around a bit more to see if this is related to venv or not. I'll report back later.
pichuan
commented
2 years ago Hi @abueg ,
two things here:
I'll keep this issue open, and we can try again with the new release soon. Thanks for reporting the issue!
abueg
commented
2 years ago Hi @pichuan , thanks for looking into this!
I tried pip install .[cpu] with a new venv and the installation seems to have worked, but then run_all_tests failed much more quickly, I'll put the output at the end of this message in case it is helpful.
That being said, it probably makes more sense for me to wait for the v0.3 release, so I'll just give it a shot when that is out, thank you for the heads up! 👍🏼
Output of trying pip install .[cpu] and then run_all_tests.sh:
[labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ python3 -m venv $SCRATCH/venvs/deepconsensus_venv_8
[labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ source $SCRATCH/venvs/deepconsensus_venv_8/bin/activate
(deepconsensus_venv_8) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_8/bin/python3 -m pip install --upgrade pip
Collecting pip
Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
Successfully uninstalled pip-20.2.3
Successfully installed pip-22.1.2
(deepconsensus_venv_8) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ pip install .[cpu]
Processing /lustre/fs5/vgl/scratch/labueg/deepconsensus
Preparing metadata (setup.py) ... done
Collecting numpy>=1.19
Using cached numpy-1.23.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Collecting pandas>=1.1
Using cached pandas-1.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
Collecting tf-models-official<=2.7.0,>=2.4.0
Using cached tf_models_official-2.7.0-py2.py3-none-any.whl (1.8 MB)
Collecting ml_collections>=0.1.0
Using cached ml_collections-0.1.1.tar.gz (77 kB)
Preparing metadata (setup.py) ... done
Collecting absl-py>=0.13.0
Using cached absl_py-1.1.0-py3-none-any.whl (123 kB)
Collecting pysam==0.19.0
Using cached pysam-0.19.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (16.3 MB)
Collecting intel-tensorflow<=2.7.0,>=2.4.0
Using cached intel_tensorflow-2.7.0-cp38-cp38-manylinux2010_x86_64.whl (186.4 MB)
Collecting astunparse>=1.6.0
Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting typing-extensions>=3.6.6
Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting tensorflow-io-gcs-filesystem>=0.21.0
Using cached tensorflow_io_gcs_filesystem-0.26.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)
Collecting keras-preprocessing>=1.1.1
Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting gast<0.5.0,>=0.2.1
Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting wrapt>=1.11.0
Using cached wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (81 kB)
Collecting h5py>=2.9.0
Using cached h5py-3.7.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
Collecting google-pasta>=0.1.1
Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting libclang>=9.0.1
Using cached libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)
Collecting flatbuffers<3.0,>=1.12
Using cached flatbuffers-2.0-py2.py3-none-any.whl (26 kB)
Collecting termcolor>=1.1.0
Using cached termcolor-1.1.0.tar.gz (3.9 kB)
Preparing metadata (setup.py) ... done
Collecting protobuf>=3.9.2
Using cached protobuf-4.21.2-cp37-abi3-manylinux2014_x86_64.whl (407 kB)
Collecting opt-einsum>=2.3.2
Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting grpcio<2.0,>=1.24.3
Using cached grpcio-1.47.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
Collecting tensorboard~=2.6
Using cached tensorboard-2.9.1-py3-none-any.whl (5.8 MB)
Collecting tensorflow-estimator<2.8,~=2.7.0rc0
Using cached tensorflow_estimator-2.7.0-py2.py3-none-any.whl (463 kB)
Collecting six>=1.12.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting wheel<1.0,>=0.32.0
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting keras<2.8,>=2.7.0rc0
Using cached keras-2.7.0-py2.py3-none-any.whl (1.3 MB)
Collecting PyYAML
Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Collecting contextlib2
Using cached contextlib2-21.6.0-py2.py3-none-any.whl (13 kB)
Collecting pytz>=2020.1
Using cached pytz-2022.1-py2.py3-none-any.whl (503 kB)
Collecting python-dateutil>=2.8.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting sentencepiece
Using cached sentencepiece-0.1.96-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting tf-slim>=1.1.0
Using cached tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)
Collecting Cython
Using cached Cython-0.29.30-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
Collecting Pillow
Using cached Pillow-9.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting tensorflow-model-optimization>=0.4.1
Using cached tensorflow_model_optimization-0.7.2-py2.py3-none-any.whl (237 kB)
Collecting tensorflow-addons
Using cached tensorflow_addons-0.17.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting tensorflow-text>=2.7.0
Using cached tensorflow_text-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
Collecting seqeval
Using cached seqeval-1.2.2.tar.gz (43 kB)
Preparing metadata (setup.py) ... done
Collecting scipy>=0.19.1
Using cached scipy-1.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.6 MB)
Collecting kaggle>=1.3.9
Using cached kaggle-1.5.12.tar.gz (58 kB)
Preparing metadata (setup.py) ... done
Collecting opencv-python-headless
Using cached opencv_python_headless-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (48.3 MB)
Collecting google-api-python-client>=1.6.7
Using cached google_api_python_client-2.52.0-py2.py3-none-any.whl (8.7 MB)
Collecting tensorflow>=2.7.0
Using cached tensorflow-2.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
Collecting py-cpuinfo>=3.3.0
Using cached py-cpuinfo-8.0.0.tar.gz (99 kB)
Preparing metadata (setup.py) ... done
Collecting gin-config
Using cached gin_config-0.5.0-py3-none-any.whl (61 kB)
Collecting oauth2client
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting psutil>=5.4.3
Using cached psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (284 kB)
Collecting pycocotools
Using cached pycocotools-2.0.4-cp38-cp38-linux_x86_64.whl
Collecting sacrebleu
Using cached sacrebleu-2.1.0-py3-none-any.whl (92 kB)
Collecting tensorflow-hub>=0.6.0
Using cached tensorflow_hub-0.12.0-py2.py3-none-any.whl (108 kB)
Collecting matplotlib
Using cached matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
Collecting tensorflow-datasets
Using cached tensorflow_datasets-4.6.0-py3-none-any.whl (4.3 MB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.8.2-py3-none-any.whl (114 kB)
Collecting google-auth-httplib2>=0.1.0
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<5,>=3.0.1
Using cached uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting google-auth<3.0.0dev,>=1.19.0
Using cached google_auth-2.9.0-py2.py3-none-any.whl (167 kB)
Collecting httplib2<1dev,>=0.15.0
Using cached httplib2-0.20.4-py3-none-any.whl (96 kB)
Collecting certifi
Using cached certifi-2022.6.15-py3-none-any.whl (160 kB)
Collecting requests
Downloading requests-2.28.1-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 7.1 MB/s eta 0:00:00
Collecting tqdm
Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting python-slugify
Using cached python_slugify-6.1.2-py2.py3-none-any.whl (9.4 kB)
Collecting urllib3
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Requirement already satisfied: setuptools>=41.0.0 in /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_8/lib/python3.8/site-packages (from tensorboard~=2.6->intel-tensorflow<=2.7.0,>=2.4.0->deepconsensus==0.2.0) (49.2.1)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
Collecting werkzeug>=1.0.1
Using cached Werkzeug-2.1.2-py3-none-any.whl (224 kB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting protobuf>=3.9.2
Using cached protobuf-3.19.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting markdown>=2.6.8
Using cached Markdown-3.3.7-py3-none-any.whl (97 kB)
Collecting tensorflow>=2.7.0
Using cached tensorflow-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
Using cached tensorflow-2.8.2-cp38-cp38-manylinux2010_x86_64.whl (498.0 MB)
Collecting tensorboard~=2.6
Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 49.4 MB/s eta 0:00:00
Collecting tensorflow>=2.7.0
Using cached tensorflow-2.8.1-cp38-cp38-manylinux2010_x86_64.whl (498.0 MB)
Using cached tensorflow-2.8.0-cp38-cp38-manylinux2010_x86_64.whl (497.6 MB)
Collecting tf-estimator-nightly==2.8.0.dev2021122109
Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.5/462.5 kB 16.8 MB/s eta 0:00:00
Collecting tensorflow>=2.7.0
Using cached tensorflow-2.7.3-cp38-cp38-manylinux2010_x86_64.whl (495.5 MB)
Collecting dm-tree~=0.1.1
Using cached dm_tree-0.1.7-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (142 kB)
INFO: pip is looking at multiple versions of tensorflow-text to determine which version is compatible with other requirements. This could take a while.
Collecting tensorflow-text>=2.7.0
Using cached tensorflow_text-2.8.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.9 MB)
Using cached tensorflow_text-2.8.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.9 MB)
Using cached tensorflow_text-2.7.3-cp38-cp38-manylinux2010_x86_64.whl (4.9 MB)
Collecting fonttools>=4.22.0
Using cached fonttools-4.33.3-py3-none-any.whl (930 kB)
Collecting cycler>=0.10
Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting packaging>=20.0
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting kiwisolver>=1.0.1
Using cached kiwisolver-1.4.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting pyparsing>=2.2.1
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting portalocker
Using cached portalocker-2.4.0-py2.py3-none-any.whl (16 kB)
Collecting regex
Using cached regex-2022.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (764 kB)
Collecting tabulate>=0.8.9
Using cached tabulate-0.8.10-py3-none-any.whl (29 kB)
Collecting colorama
Using cached colorama-0.4.5-py2.py3-none-any.whl (16 kB)
Collecting scikit-learn>=0.21.3
Using cached scikit_learn-1.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31.2 MB)
Collecting typeguard>=2.7
Using cached typeguard-2.13.3-py3-none-any.whl (17 kB)
Collecting etils[epath]
Using cached etils-0.6.0-py3-none-any.whl (98 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting importlib-resources
Using cached importlib_resources-5.8.0-py3-none-any.whl (28 kB)
Collecting dill
Using cached dill-0.3.5.1-py2.py3-none-any.whl (95 kB)
Collecting promise
Using cached promise-2.3.tar.gz (19 kB)
Preparing metadata (setup.py) ... done
Collecting tensorflow-metadata
Using cached tensorflow_metadata-1.9.0-py3-none-any.whl (51 kB)
Collecting googleapis-common-protos<2.0dev,>=1.56.2
Using cached googleapis_common_protos-1.56.3-py2.py3-none-any.whl (211 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
Using cached importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Collecting charset-normalizer<3,>=2
Downloading charset_normalizer-2.1.0-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting joblib>=1.0.0
Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting zipp
Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting text-unidecode>=1.3
Using cached text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Using legacy 'setup.py install' for ml_collections, since package 'wheel' is not installed.
Using legacy 'setup.py install' for deepconsensus, since package 'wheel' is not installed.
Using legacy 'setup.py install' for kaggle, since package 'wheel' is not installed.
Using legacy 'setup.py install' for py-cpuinfo, since package 'wheel' is not installed.
Using legacy 'setup.py install' for termcolor, since package 'wheel' is not installed.
Using legacy 'setup.py install' for seqeval, since package 'wheel' is not installed.
Using legacy 'setup.py install' for promise, since package 'wheel' is not installed.
Installing collected packages: text-unidecode, termcolor, tensorflow-estimator, tensorboard-plugin-wit, sentencepiece, pytz, pysam, pyasn1, py-cpuinfo, libclang, keras, gin-config, flatbuffers, dm-tree, zipp, wrapt, wheel, werkzeug, urllib3, uritemplate, typing-extensions, typeguard, tqdm, toml, threadpoolctl, tensorflow-io-gcs-filesystem, tensorboard-data-server, tabulate, six, rsa, regex, PyYAML, python-slugify, pyparsing, pyasn1-modules, psutil, protobuf, portalocker, Pillow, oauthlib, numpy, kiwisolver, joblib, idna, gast, fonttools, etils, dill, Cython, cycler, contextlib2, colorama, charset-normalizer, certifi, cachetools, absl-py, tf-slim, tensorflow-model-optimization, tensorflow-hub, scipy, sacrebleu, requests, python-dateutil, promise, packaging, opt-einsum, opencv-python-headless, ml_collections, keras-preprocessing, importlib-resources, importlib-metadata, httplib2, h5py, grpcio, googleapis-common-protos, google-pasta, google-auth, astunparse, tensorflow-metadata, tensorflow-addons, scikit-learn, requests-oauthlib, pandas, oauth2client, matplotlib, markdown, kaggle, google-auth-httplib2, google-api-core, tensorflow-datasets, seqeval, pycocotools, google-auth-oauthlib, google-api-python-client, tensorboard, tensorflow, intel-tensorflow, tensorflow-text, tf-models-official, deepconsensus
Running setup.py install for termcolor ... done
Running setup.py install for py-cpuinfo ... done
Running setup.py install for promise ... done
Running setup.py install for ml_collections ... done
Running setup.py install for kaggle ... done
Running setup.py install for seqeval ... done
Running setup.py install for deepconsensus ... done
Successfully installed Cython-0.29.30 Pillow-9.1.1 PyYAML-6.0 absl-py-1.1.0 astunparse-1.6.3 cachetools-5.2.0 certifi-2022.6.15 charset-normalizer-2.1.0 colorama-0.4.5 contextlib2-21.6.0 cycler-0.11.0 deepconsensus-0.2.0 dill-0.3.5.1 dm-tree-0.1.7 etils-0.6.0 flatbuffers-2.0 fonttools-4.33.3 gast-0.4.0 gin-config-0.5.0 google-api-core-2.8.2 google-api-python-client-2.52.0 google-auth-2.9.0 google-auth-httplib2-0.1.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 googleapis-common-protos-1.56.3 grpcio-1.47.0 h5py-3.7.0 httplib2-0.20.4 idna-3.3 importlib-metadata-4.12.0 importlib-resources-5.8.0 intel-tensorflow-2.7.0 joblib-1.1.0 kaggle-1.5.12 keras-2.7.0 keras-preprocessing-1.1.2 kiwisolver-1.4.3 libclang-14.0.1 markdown-3.3.7 matplotlib-3.5.2 ml_collections-0.1.1 numpy-1.23.0 oauth2client-4.1.3 oauthlib-3.2.0 opencv-python-headless-4.6.0.66 opt-einsum-3.3.0 packaging-21.3 pandas-1.4.3 portalocker-2.4.0 promise-2.3 protobuf-3.19.4 psutil-5.9.1 py-cpuinfo-8.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycocotools-2.0.4 pyparsing-3.0.9 pysam-0.19.0 python-dateutil-2.8.2 python-slugify-6.1.2 pytz-2022.1 regex-2022.6.2 requests-2.28.1 requests-oauthlib-1.3.1 rsa-4.8 sacrebleu-2.1.0 scikit-learn-1.1.1 scipy-1.8.1 sentencepiece-0.1.96 seqeval-1.2.2 six-1.16.0 tabulate-0.8.10 tensorboard-2.9.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.7.3 tensorflow-addons-0.17.1 tensorflow-datasets-4.6.0 tensorflow-estimator-2.7.0 tensorflow-hub-0.12.0 tensorflow-io-gcs-filesystem-0.26.0 tensorflow-metadata-1.9.0 tensorflow-model-optimization-0.7.2 tensorflow-text-2.7.3 termcolor-1.1.0 text-unidecode-1.3 tf-models-official-2.7.0 tf-slim-1.1.0 threadpoolctl-3.1.0 toml-0.10.2 tqdm-4.64.0 typeguard-2.13.3 typing-extensions-4.2.0 uritemplate-4.1.1 urllib3-1.26.9 werkzeug-2.1.2 wheel-0.37.1 wrapt-1.14.1 zipp-3.8.0
(deepconsensus_venv_8) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ l
total 5.9M
drwxr-xr-x 7 labueg jarv 4.0K Jun 29 10:50 ./
drwxr-xr-x 41 labueg jarv 4.0K Jun 27 16:05 ../
drwxr-xr-x 4 labueg jarv 4.0K Jun 29 10:48 build/
-rw-r--r-- 1 labueg jarv 850 Jun 27 16:05 CONTRIBUTING.md
drwxr-xr-x 9 labueg jarv 4.0K Jun 27 16:08 deepconsensus/
drwxr-xr-x 2 labueg jarv 4.0K Jun 29 11:17 deepconsensus.egg-info/
-rw-r--r-- 1 labueg jarv 1.8K Jun 27 16:05 Dockerfile
drwxr-xr-x 3 labueg jarv 4.0K Jun 27 16:05 docs/
drwxr-xr-x 8 labueg jarv 4.0K Jun 27 16:05 .git/
-rw-r--r-- 1 labueg jarv 2.5K Jun 28 13:07 install_edit.sh
-rw-r--r-- 1 labueg jarv 2.6K Jun 27 16:05 install-gpu.sh
-rw-r--r-- 1 labueg jarv 2.6K Jun 27 16:05 install.sh
-rw-r--r-- 1 labueg jarv 1.5K Jun 27 16:05 LICENSE
-rw-r--r-- 1 labueg jarv 2.2K Jun 27 16:05 README.md
-rw-r--r-- 1 labueg jarv 595 Jun 27 16:05 README_pip.md
-rw-r--r-- 1 labueg jarv 111 Jun 28 13:03 requirements.txt
-rwxr-xr-x 1 labueg jarv 2.3K Jun 27 16:05 run_all_tests.sh*
-rw-r--r-- 1 labueg jarv 3.0K Jun 27 16:05 setup.py
-rw-r--r-- 1 labueg jarv 1.5M Jun 27 17:55 slurm-28038284.out
-rw-r--r-- 1 labueg jarv 1.5M Jun 28 13:32 slurm-28054017.out
-rw-r--r-- 1 labueg jarv 1.5M Jun 28 14:24 slurm-28056114.out
-rw-r--r-- 1 labueg jarv 1.5M Jun 29 10:56 slurm-28415929.out
(deepconsensus_venv_8) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ sb ./run_all_tests.sh
Submitted batch job 28433686
(deepconsensus_venv_8) [labueg@login04 /lustre/fs5/vgl/scratch/labueg/deepconsensus]$ cat slurm-28433686.out
Running tests under Python 3.8.8: /lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_8/bin/python3
[ RUN ] QuickInferenceTest.test_end_to_end0 (subreads='human_1m/subreads_to_ccs.bam', fasta='human_1m/ccs.fasta', expected_lengths=[17141, 16320])
2022-06-29 11:24:02.764789: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-06-29 11:24:02.779965: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
[E::idx_find_and_load] Could not retrieve index file for 'deepconsensus/testdata/human_1m/subreads_to_ccs.bam'
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
[ FAILED ] QuickInferenceTest.test_end_to_end0 (subreads='human_1m/subreads_to_ccs.bam', fasta='human_1m/ccs.fasta', expected_lengths=[17141, 16320])
[ RUN ] QuickInferenceTest.test_end_to_end_multiprocessing0 (cpus=0, batch_zmws=1)
[E::idx_find_and_load] Could not retrieve index file for 'deepconsensus/testdata/human_1m/subreads_to_ccs.bam'
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
[ OK ] QuickInferenceTest.test_end_to_end_multiprocessing0 (cpus=0, batch_zmws=1)
[ RUN ] QuickInferenceTest.test_end_to_end_multiprocessing1 (cpus=0, batch_zmws=0)
[E::idx_find_and_load] Could not retrieve index file for 'deepconsensus/testdata/human_1m/subreads_to_ccs.bam'
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
[ OK ] QuickInferenceTest.test_end_to_end_multiprocessing1 (cpus=0, batch_zmws=0)
[ RUN ] QuickInferenceTest.test_end_to_end_multiprocessing2 (cpus=1, batch_zmws=1)
[E::idx_find_and_load] Could not retrieve index file for 'deepconsensus/testdata/human_1m/subreads_to_ccs.bam'
expected lengths: [17141, 16320] output lengths: [16994, 16102]
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
[ OK ] QuickInferenceTest.test_end_to_end_multiprocessing2 (cpus=1, batch_zmws=1)
[ RUN ] QuickInferenceTest.test_end_to_end_multiprocessing3 (cpus=1, batch_zmws=100)
[E::idx_find_and_load] Could not retrieve index file for 'deepconsensus/testdata/human_1m/subreads_to_ccs.bam'
/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/preprocess/utils.py:134: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.seq_indices = np.zeros(len(self.bases), dtype=np.int)
[ OK ] QuickInferenceTest.test_end_to_end_multiprocessing3 (cpus=1, batch_zmws=100)
======================================================================
FAIL: test_end_to_end0 (subreads='human_1m/subreads_to_ccs.bam', fasta='human_1m/ccs.fasta', expected_lengths=[17141, 16320]) (__main__.QuickInferenceTest)
QuickInferenceTest.test_end_to_end0 (subreads='human_1m/subreads_to_ccs.bam', fasta='human_1m/ccs.fasta', expected_lengths=[17141, 16320])
test_end_to_end(subreads='human_1m/subreads_to_ccs.bam', fasta='human_1m/ccs.fasta', expected_lengths=[17141, 16320])
----------------------------------------------------------------------
Traceback (most recent call last):
File "/lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_8/lib/python3.8/site-packages/absl/testing/parameterized.py", line 312, in bound_param_test
return test_method(self, **testcase_params)
File "/lustre/fs5/vgl/scratch/labueg/venvs/deepconsensus_venv_8/lib/python3.8/site-packages/absl/testing/flagsaver.py", line 150, in _flagsaver_wrapper
return func(*args, **kwargs)
File "/lustre/fs5/vgl/scratch/labueg/deepconsensus/deepconsensus/inference/quick_inference_test.py", line 83, in test_end_to_end
self.assertEqual(outcomes.success, 2)
AssertionError: 0 != 2
----------------------------------------------------------------------
Ran 5 tests in 20.007s
FAILED (failures=1)For more context about the machine in case it is helpful here is its information:
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.8 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"Hi @pichuan , congratulations on the v0.3 release! 🎉
I did give installing with this new release a try (using same method as before with a venv for deepconsensus, and then a conda env for the pacbio dependencies etc), and it seems to have worked fine.
run_all_tests looked fine (all tests OK) outside of some warnings, but nothing errorred out. And I tried with the test data in the quick start documentation, and that also looks good:
Total params: 8,942,667
Trainable params: 8,942,667
Non-trainable params: 0
_________________________________________________________________
I0707 18:05:15.126298 140418158692160 model_utils.py:231] Setting hidden size to transformer_input_size.
I0707 18:05:15.126487 140418158692160 quick_inference.py:484] Finished initialize_model.
I0707 18:05:15.127358 140418158692160 quick_inference.py:738] Model setup took 0.8111944198608398 seconds.
I0707 18:06:47.645821 140418158692160 quick_inference.py:623] Example summary: ran model=21414 (66.91%; 74.461s) skip=10591 (33.09%; 1.469s) total=32005.
I0707 18:06:48.601975 140418158692160 quick_inference.py:662] Processed a batch of 100 ZMWs in 81.998 seconds
I0707 18:06:48.683865 140418158692160 quick_inference.py:779] Processed 100 ZMWs in 93.556 seconds
I0707 18:08:08.612023 140418158692160 quick_inference.py:623] Example summary: ran model=18669 (71.82%; 65.191s) skip=7326 (28.18%; 1.189s) total=25995.
I0707 18:08:09.388492 140418158692160 quick_inference.py:662] Processed a batch of 78 ZMWs in 71.374 seconds
I0707 18:08:09.509874 140418158692160 quick_inference.py:798] Processed 178 ZMWs in 174.382 seconds
I0707 18:08:09.510046 140418158692160 quick_inference.py:800] Outcome counts: OutcomeCounter(empty_sequence=0, only_gaps_and_padding=0, failed_quality_filter=0, failed_length_filter=0, success=178)Thank you so much for your feedback before, I'm excited to work with this promising tool! (Feel free to close this issue, just wanted to give you an update 👍🏼 )
pichuan
commented
2 years ago Thank you for the update! Glad to hear that it works now!
Hello! :wave:
I am trying to install DeepConsensus on an HPC environment (no GPU) without root permissions, and without Docker/Singularity access. I am approaching this by making a deepconsensus venv and trying to install with a modified install script. I have done the following steps: 1)
python3 -m venv $SCRATCH/venvs/deepconsensus_venv_12)source deepconsensus_venv_1/bin/activate3)source install_edit.sh, whereinstall_edit.shskips the apt-get steps and just upgrades pip + installs requirements.txt & intel-tensorflow for the venvHere are the contents of install_edit.sh:
And here is the output from running that install script:
I then ran
run_all_tests.shon a compute node via slurm and the full output is in this gist: https://gist.github.com/abueg/421ca972563b5c32825cde17525a49bfThere is this exception in the output, which leads me to think the installation failed:
Would this error arise from the tensorflow dependency problem, in which case should I try to install tensorflow 2.9.1? Or have I gone wrong somewhere else?
There is also this error before the exception:
[E::idx_find_and_load] Could not retrieve index file for 'deepconsensus/testdata/human_1m/subreads_to_ccs.bam', but there is no index file for that file in thedeepconsensus/testdata/human_1m/directory, so should I index it prior to running the tests?Any help would be appreciated, thank you in advance!