PopicLab / cue

Deep learning framework for SV calling and genotyping
MIT License
102 stars 20 forks source link

Tutorial Error #22

Closed jkalleberg closed 1 year ago

jkalleberg commented 1 year ago

Hello,

I'm sorry in advance for the wall of text; however, I've repeatedly tried to build the environment without success and wanted to be thorough for reproducibility. I would greatly appreciate any help!

To avoid using long local paths, use these paths as examples: I pulled the git repo to /root/path/cue and ran the tutorial at WORKING_DIR=/root/path/project_dirname.

I am getting the following error while trying to run the tutorial:

Output (edited to have ${WORKING_DIR} instead of the long absolute path):

*********************************
*  cue (v0.2.2): discovery mode *
*********************************
[INFO]  ========== Model config ==========
        model_path: ${WORKING_DIR}/data/demo/models/cue.pt
        n_cpus: 1
        gpu_ids: []
        batch_size: 16
        logging_level: INFO
        report_interval: 50
        n_jobs_per_gpu: 1
        signal_set: SV_SIGNAL_SET.SHORT
        class_set: SV_CLASS_SET.BASIC5ZYG
        num_keypoints: 1
        model_architecture: HG
        image_dim: 256
        sigma: 10
        stride: 4
        heatmap_peak_threshold: 0.4
        pretrained_refinenn_path: None
        config_file: ${WORKING_DIR}/data/demo/results/model.yaml
        experiment_dir: ${WORKING_DIR}/data/demo/results
        devices: [device(type='cpu')]
        device: cpu
        log_dir: ${WORKING_DIR}/data/demo/results/logs/
        report_dir: ${WORKING_DIR}/data/demo/results/reports/
        log_file: ${WORKING_DIR}/data/demo/results/logs/main.log
        classes: ['NEG', 'DEL-HOM', 'INV-HOM', 'DUP-HOM', 'DEL-HET', 'INV-HET', 'DUP-HET', 'IDUP-HOM', 'IDUP-HET']
        num_classes: 9
        n_signals: 6
[INFO] ========== Data config =========
        bam: ${WORKING_DIR}/data/demo/inputs/chr21.small.bam
        fai: ${WORKING_DIR}/data/demo/inputs/GRCh38.fa.fai
        chr_names: ['chr21']
        n_cpus: 1
        logging_level: INFO
        min_refine_buffer: 2000
        refine_buffer_frac_size: 5
        refine_pair_dist_frac_size: 2
        refine_bp_kernels: [0, 50, 500]
        refine_min_support: 2
        refine_disable: False
        min_pair_support: 2
        min_pair_distance: 4000
        max_pair_distance: 1000000
        scan_target_intervals: True
        stream: True
        view_mode: False
        store_img: False
        empty_annotation: False
        bins_per_block: 8000
        min_sv_len: 4000
        min_qual_score: 50
        bam_type: BAM_TYPE.SHORT
        signal_set: SV_SIGNAL_SET.SHORT
        signal_set_origin: SHORT
        bed: None
        blacklist_bed: None
        signal_vmax: {'RD': 600, 'RD_LOW': 800, 'RD_CLIPPED': 600, 'SM': 200, 'SR_RP': 600, 'LR': 600, 'LLRR': 100, 'RL': 100, 'LLRR_VS_LR': 1}
        signal_mapq: {'RD': 20, 'RD_LOW': 0, 'RD_CLIPPED': 20, 'SM': 20, 'SR_RP': 0, 'LR': 0, 'LLRR': 1, 'RL': 1, 'LLRR_VS_LR': 1}
        bin_size: 750
        interval_size: 150000
        step_size: 50000
        shift_size: None
        heatmap_dim: 1000
        image_dim: 256
        class_set: SV_CLASS_SET.BASIC5ZYG
        num_keypoints: 1
        bbox_padding: 0
        config_file: ${WORKING_DIR}/data/demo/results/data.yaml
        dataset_dir: ${WORKING_DIR}/data/demo/results
        info_dir: ${WORKING_DIR}/data/demo/results/info/
        image_dir: ${WORKING_DIR}/data/demo/results/images/
        annotation_dir: ${WORKING_DIR}/data/demo/results/annotations/
        annotated_images_dir:${WORKING_DIR}/data/demo/results/annotated_images/
        classes: ['NEG', 'DEL-HOM', 'INV-HOM', 'DUP-HOM', 'DEL-HET', 'INV-HET', 'DUP-HET', 'IDUP-HOM', 'IDUP-HET']
        num_classes: 9
        num_signals: 6
        uid: 0000000000
        log_file: ${WORKING_DIR}/data/demo/results/info/main.log
[INFO] Running on 1 CPUs/GPUs
[INFO] Chromosome lists processed by each process: [array(['chr21'], dtype='<U5')]
Traceback (most recent call last):
  File "../cue/engine/call.py", line 105, in <module>
    delayed(call)(config.devices[i], chr_name_chunks[i], scan_id) for i in range(n_procs))
  File "${WORKING_DIR}/miniconda_envs/dev/lib/python3.7/site-packages/joblib/parallel.py", line 1855, in __call__
    return output if self.return_generator else list(output)
  File "${WORKING_DIR}/miniconda_envs/dev/lib/python3.7/site-packages/joblib/parallel.py", line 1784, in _get_sequential_output
    res = func(*args, **kwargs)
  File "../cue/engine/call.py", line 73, in call  
    model.load_state_dict(torch.load(config.model_path, device))
  File "${WORKING_DIR}/miniconda_envs/dev/lib/python3.7/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "${WORKING_DIR}/miniconda_envs/dev/lib/python3.7/site-packages/torch/serialization.py", line 763, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'. 

Tutorial Bash Script

#!/bin/bash
# scripts/setup/tutorial.sh

echo -e "=== scripts/setup/tutorial.sh > start $(date)"

# add directories if missing
mkdir -p data/demo/inputs
mkdir -p data/demo/ground_truth
mkdir -p data/demo/models
mkdir -p data/demo/results

# copy tutorial raw data files
CUE_DIR=https://storage.googleapis.com/cue-demo/

declare -a FILES=("inputs/chr21.small.bam" "inputs/chr21.small.bam.bai" "inputs/GRCh38.fa.fai" "ground_truth/svs.chr21.small.sorted.vcf.gz" "ground_truth/svs.chr21.small.sorted.vcf.gz.tbi" "models/cue.pt")

for f in ${FILES[@]}; do
    if [ ! -f ./data/demo/${f} ]; then
        echo "$(date '+%Y-%m-%d %H:%M:%S') INFO: downloading a file now... | '${CUE_DIR}${f}'"
        curl -s --continue-at - ${CUE_DIR}/$f -o ./data/demo/$f
    else
        echo "$(date '+%Y-%m-%d %H:%M:%S') INFO: file found | './data/demo/${f}'"
    fi
done

# copy configuration files for demo
cp ../cue/data/demo/config/data.yaml data/demo/results/.
cp ../cue/data/demo/config/model.yaml data/demo/results/.

export WORKING_DIR=$(pwd)
## THEN, manually edit copied files to replace '..' with 'WORKING_DIR' path 

# Activate the environment -----------------
conda activate ./miniconda_envs/dev/

# Adding path to Cue -----------------------
ROOT_DIR=$(dirname $(pwd))
export PYTHONPATH=${ROOT_DIR}/cue:${PYTHONPATH}

python ../cue/engine/call.py --data_config ${WORKING_DIR}/data/demo/results/data.yaml --model_config ${WORKING_DIR}/data/demo/results/model.yaml

I initially assumed my package/dependency versions were introducing a syntax error. However, being explicit using version numbers from the requirements.txt was unsuccessful. Instead, I built the conda env repeatedly, landing on success by specifying Python3.7 and torch v0.6 using the following bash script, but I still get the error:

#!/bin/bash
# scripts/setup/build_env.sh

echo -e "=== scripts/setup/build_env.sh > start $(date)"

##--- NOTE: ----##
##  You must have an interactive session
##  with more mem than defaults to work!
##--------------##

if [ ! -d ./miniconda_envs/dev ] ; then
     # If missing an enviornment called "dev", 
     # initalize this env with only the anaconda package 
     conda create --yes --prefix ./miniconda_envs/dev
fi

# Then, activate the base environment to enable 'conda activate'
source ${CONDA_BASE}/etc/profile.d/conda.sh
conda deactivate

##--- Configure an environment-specific .condarc file ---##
## NOTE: Only performed once:
# Changes the (env) prompt to avoid printing the full path
conda config --env --set env_prompt '({name})'

# Put the package download channels in a specific order
conda config --env --add channels defaults
conda config --env --add channels bioconda
conda config --env --add channels conda-forge

# Download packages flexibly
conda config --env --set channel_priority flexible

# Install the project-specific packages in the env
conda install -p ./miniconda_envs/dev -y pip python=3.7 numpy scipy pyfaidx pysam pytabix truvari opencv

conda install -p ./miniconda_envs/dev -y bitarray cachetools intervaltree joblib matplotlib pycocotools python-dateutil pyyaml seaborn

conda install -p ./miniconda_envs/dev -y python-dotenv regex natsort mkdocs mkdocs-material black 

# using a private channel to obtain outdated versions required by Cue
conda install -p ./miniconda_envs/dev -y -c zeus1942 torchvision=0.6 pytorch

###===== Notes about specific packages =====###
### Python = Cue docs specify Python3.7
### Scipy = scientific libraries for Python
### DotEnv = enables environment variable configuration across bash and python
### Regex = required for update regular expression handling
### Natsort = enables sorting of file iterators
### Mkdocs & Mkdocs-Material = used for writing Github documentation
### Black = Python formatter
### REMAINING PACKAGES ARE CUE REQUIREMENTS OBTAINED FROM | https://github.com/PopicLab/cue/blob/551cb72f7b9f4177c9ac71743f8ab7cf3d7f28dc/install/requirements.txt

echo -e "=== scripts/setup/build_env.sh > end $(date)"

My conda environment includes the following:

(dev)[jakth2@lewis4-c8k-hpc2-node278 scripts]$ conda --version
conda 4.11.0
(dev)[jakth2@lewis4-c8k-hpc2-node278 scripts]$ conda list
# packages in environment at /storage/hpc/group/UMAG_test/WORKING/jakth2/SVS_230718/miniconda_envs/dev:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
aom                       3.5.0                h27087fc_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
biopython                 1.79             py37h540881e_2    conda-forge
bitarray                  2.6.0            py37h540881e_0    conda-forge
black                     22.6.0             pyhd8ed1ab_0    conda-forge
blas                      1.0                         mkl
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotli-python             1.0.9            py37hd23a5d3_7    conda-forge
bwapy                     0.1.4            py37ha26db04_5    bioconda
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cachetools                5.3.1              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py37h43b0acd_1    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3            py37h89c1867_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
commonmark                0.9.1                      py_0    conda-forge
cudatoolkit               10.2.89             hdec6ad0_12    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    0.29.32          py37hd23a5d3_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
ffmpeg                    5.1.2           gpl_h8dda1f0_106    conda-forge
fftw                      3.3.10          nompi_hc118613_108    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.38.0           py37h540881e_0    conda-forge
freeglut                  3.2.2                h9c3ff4c_1    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
future                    0.18.2           py37h89c1867_5    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
ghp-import                2.1.0              pyhd8ed1ab_0    conda-forge
glib                      2.76.4               hfc55251_0    conda-forge
glib-tools                2.76.4               hfc55251_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.7.8                hf3e180e_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gst-plugins-base          1.21.3               h4243ec0_1    conda-forge
gstreamer                 1.21.3               h25f0c4b_1    conda-forge
gstreamer-orc             0.4.34               hd590300_0    conda-forge
harfbuzz                  5.3.0                h418a68e_0    conda-forge
hdf5                      1.12.2          nompi_h2386368_101    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py37h89c1867_0    conda-forge
intel-openmp              2021.4.0          h06a4308_3561
intervaltree              3.1.0              pyhd8ed1ab_1    conda-forge
jack                      1.9.22               h11f4161_0    conda-forge
jasper                    2.0.33               h0ff4b12_1    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.0              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py37h7cecad7_0    conda-forge
krb5                      1.20.1               hf9c8cef_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libblas                   3.9.0            12_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcap                    2.66                 ha37c62d_0    conda-forge
libcblas                  3.9.0            12_linux64_mkl    conda-forge
libclang                  15.0.7          default_h7634d5b_3    conda-forge
libclang13                15.0.7          default_h9986a30_3    conda-forge
libcups                   2.3.3                h36d4200_3    conda-forge
libcurl                   7.87.0               h6312ad2_0    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libdrm                    2.4.114              h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.4               hebfc3b9_0    conda-forge
libglu                    9.0.0             he1b5a44_1001    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.4                h166bdaf_0    conda-forge
liblapack                 3.9.0            12_linux64_mkl    conda-forge
liblapacke                3.9.0            12_linux64_mkl    conda-forge
libllvm15                 15.0.7               hadd5161_1    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libopencv                 4.6.0            py37h7b66c90_5    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpciaccess              0.17                 h166bdaf_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.1                 h2baec63_3    conda-forge
libprotobuf               3.21.12              h3eb15da_0    conda-forge
libsndfile                1.2.0                hb75c966_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libsystemd0               252                  h2a991cd_0    conda-forge
libtasn1                  4.19.0               h166bdaf_0    conda-forge
libtiff                   4.4.0                h0e0dad5_3    conda-forge
libtool                   2.4.7                h27087fc_0    conda-forge
libudev1                  253                  h0b41bf4_0    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libva                     2.18.0               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp-base              1.3.1                hd590300_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.5.0                h79f4944_0    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markdown                  3.4.4              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py37h7f8727e_0
matplotlib                3.5.3            py37h89c1867_2    conda-forge
matplotlib-base           3.5.3            py37hf395dca_2    conda-forge
mergedeep                 1.3.4              pyhd8ed1ab_0    conda-forge
mkdocs                    1.5.2              pyhd8ed1ab_0    conda-forge
mkdocs-material           9.1.21             pyhd8ed1ab_1    conda-forge
mkdocs-material-extensions 1.1.1              pyhd8ed1ab_0    conda-forge
mkl                       2021.4.0           h06a4308_640
more-itertools            9.1.0              pyhd8ed1ab_0    conda-forge
mpg123                    1.31.3               hcb278e6_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge
mysql-common              8.0.32               h14678bc_0    conda-forge
mysql-libs                8.0.32               h54cf53e_0    conda-forge
natsort                   8.4.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nettle                    3.8.1                hc379101_1    conda-forge
ninja                     1.11.1               h924138e_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numpy                     1.21.6           py37h976b520_0    conda-forge
opencv                    4.6.0            py37h89c1867_5    conda-forge
openh264                  2.3.1                hcb278e6_2    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1v               hd590300_0    conda-forge
p11-kit                   0.24.1               hc5aa10d_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
pathspec                  0.11.2             pyhd8ed1ab_0    conda-forge
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pillow                    9.2.0            py37h850a105_2    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
progressbar2              4.2.0              pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pulseaudio                16.1                 h4ab2085_1    conda-forge
py-opencv                 4.6.0            py37hf05f0b3_5    conda-forge
pycocotools               2.0.4            py37hda87dfa_2    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyfaidx                   0.7.2.1            pyh7cba7a3_1    bioconda
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pymdown-extensions        10.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7           py37hf30b843_1    conda-forge
pyqt5-sip                 12.11.0          py37hd23a5d3_1    conda-forge
pysam                     0.21.0           py37hee149a5_0    bioconda
pysocks                   1.7.1            py37h89c1867_5    conda-forge
pytabix                   0.1              py37h09c1ff4_3    bioconda
python                    3.7.12          hb7a2778_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-dotenv             1.0.0              pyhd8ed1ab_0    conda-forge
python-edlib              1.3.9            py37h96cfd12_2    bioconda
python-levenshtein        0.12.2           py37h540881e_2    conda-forge
python-utils              3.6.0              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     3_cp37m    conda-forge
pytorch                   1.5.0           py3.7_cuda10.2.89_cudnn7.6.5_0    zeus1942
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyvcf3                    1.0.3              pyhdfd78af_0    bioconda
pyyaml                    6.0              py37h540881e_4    conda-forge
pyyaml-env-tag            0.1                pyhd8ed1ab_0    conda-forge
qt-main                   5.15.6               h62441b5_5    conda-forge
readline                  8.2                  h8228510_1    conda-forge
regex                     2022.7.9         py37h5eee18b_0
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rich                      12.5.1             pyhd8ed1ab_0    conda-forge
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
setuptools                59.8.0           py37h89c1867_1    conda-forge
sip                       6.7.2            py37hd23a5d3_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.42.0               h2c6b66d_0    conda-forge
statsmodels               0.13.2           py37hda87dfa_0    conda-forge
svt-av1                   1.4.1                hcb278e6_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
torchvision               0.6.0                py37_cu102    zeus1942
tornado                   6.2              py37h540881e_0    conda-forge
truvari                   3.5.0              pyhdfd78af_0    bioconda
typed-ast                 1.4.3            py37h7f8727e_1
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
unicodedata2              14.0.0           py37h540881e_1    conda-forge
urllib3                   2.0.4              pyhd8ed1ab_0    conda-forge
watchdog                  2.1.6            py37h06a4308_0
wheel                     0.41.0             pyhd8ed1ab_0    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xcb-util                  0.4.0                h166bdaf_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h166bdaf_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h166bdaf_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.4                h0b41bf4_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      0.6.0                      py_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge

I also attempted to install packages using pip within a fresh conda env using the following:

if [ ! -d ./miniconda_envs/dev ] ; then
     # If missing an enviornment called "dev", 
     # initalize this env with only the anaconda package 
     conda create --yes --prefix ./miniconda_envs/dev
fi

# Then, activate the base environment to enable 'conda activate'
source ${CONDA_BASE}/etc/profile.d/conda.sh
conda deactivate

##--- Configure an environment-specific .condarc file ---##
## NOTE: Only performed once:
# Changes the (env) prompt to avoid printing the full path
conda config --env --set env_prompt '({name})'

# Put the package download channels in a specific order
conda config --env --add channels defaults
conda config --env --add channels bioconda
conda config --env --add channels conda-forge

# Download packages flexibly
conda config --env --set channel_priority flexible

# Install the project-specific packages in the env
conda install -p ./miniconda_envs/dev -y pip python=3.7

# activate env
. scripts/start_dev.sh

pip install cython

pip --no-cache-dir install -r ../cue/install/requirements.txt

But got the following error:

${WORKING_DIR}/miniconda_envs/dev/lib/python3.7/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `
setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
AttributeError: cython_sources
      [end of output]

 note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
jkalleberg commented 1 year ago

I think I figured it out and wanted to share it with anyone else dealing with this in the future. My conda env was fine, the demo files downloaded with curl were not. I re-downloaded them using wget via

CUE_DIR="https://storage.googleapis.com/cue-demo"

declare -a FILES=("inputs/chr21.small.bam" "inputs/chr21.small.bam.bai" "inputs/GRCh38.fa.fai" "ground_truth/svs.chr21.small.sorted.vcf.gz" "ground_truth/svs.chr21.small.sorted.vcf.gz.tbi" "models/cue.pt")

for f in ${FILES[@]}; do
    if [ ! -f ./data/demo/${f} ]; then
        echo "$(date '+%Y-%m-%d %H:%M:%S') INFO: downloading a file now... | '${CUE_DIR}/${f}'"
        AWS_FILE_NAME="${CUE_DIR}/${f}"
        PREFIX=$(echo ${f} | cut -d "/" -f 1)
        echo "./data/demo/${PREFIX}"
        wget --directory-prefix="./data/demo/${PREFIX}" ${AWS_FILE_NAME} 
    else
        echo "$(date '+%Y-%m-%d %H:%M:%S') INFO: file found | './data/demo/${f}'"
    fi
done

This got me past the Unpickeling.Error but then I ran into another error below:

Traceback (most recent call last):
  File "../cue/engine/call.py", line 107, in <module>
    joblib.delayed(call)(config.devices[i], chr_name_chunks[i], scan_id) for i in range(n_procs))
  File "/storage/hpc/group/UMAG_test/WORKING/jakth2/SVS_230718/miniconda_envs/dev/lib/python3.7/site-packages/joblib/parallel.py", line 1855, in __call__
    return output if self.return_generator else list(output)
  File "/storage/hpc/group/UMAG_test/WORKING/jakth2/SVS_230718/miniconda_envs/dev/lib/python3.7/site-packages/joblib/parallel.py", line 1784, in _get_sequential_output
    res = func(*args, **kwargs)
  File "../cue/engine/call.py", line 90, in call  
    collect_data_metrics=True, given_ground_truth=given_ground_truth)
  File "/storage/hpc/group/UMAG_test/WORKING/jakth2/cue/engine/core.py", line 98, in evaluate
    targets2=targets)
  File "/storage/hpc/group/UMAG_test/WORKING/jakth2/cue/img/plotting.py", line 202, in plot_images
    image = annotate(image, targets[indices[i]], classes, display_boxes=True, color=(0, 76 / 255, 153 / 255))
  File "/storage/hpc/group/UMAG_test/WORKING/jakth2/cue/img/plotting.py", line 161, in annotate
    cv2.circle(image, (p[0], p[1]), int(image_dim/100), color=color, thickness=-thickness)
cv2.error: OpenCV(4.6.0) :-1: error: (-5:Bad argument) in function 'circle'
> Overload resolution failed:
>  - Can't parse 'center'. Sequence item with index 0 has a wrong type
>  - Can't parse 'center'. Sequence item with index 0 has a wrong type

To fix this I changed both instances of v2.circle(image, (p[0], p[1]) in plotting.py to cv2.circle(image, (int(p[0]), int(p[1])).

Finally, I used the bug fix in Issue #21 to get the tutorial to run.