KuechlerO
commented
3 months ago Ah, I've tried also the docker container. However, the same problem appears also in the docker container:
(absplice_dock) root@e39380687b11:/app/absplice/example/workflow# python -m snakemake -j 1 --use-conda
/app/absplice/example/workflow
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Creating conda environment ../envs/environment_spliceai_rocksdb.yaml...
Downloading and installing remote packages.
Environment for /app/absplice/example/workflow/splicing_pred/DNA/../../../envs/environment_spliceai_rocksdb.yaml created (location: .snakemake/conda/9241678ce9daadc3d4184a0af19626a0_)
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
-------------------- ------- ------------- -------------
absplice_dna 1 1 1
all 1 1 1
download_human_fasta 1 1 1
download_splicemaps 1 1 1
mmsplice_splicemap 1 1 1
spliceai 1 1 1
spliceai_vcf_to_csv 1 1 1
total 7 1 1
Select jobs to execute...
[Fri May 24 11:08:02 2024]
rule download_splicemaps:
output: ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi3_method=kn_event_filter=median_cutoff.csv.gz, ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi5_method=kn_event_filter=median_cutoff.csv.gz
jobid: 2
reason: Missing output files: ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi3_method=kn_event_filter=median_cutoff.csv.gz, ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi5_method=kn_event_filter=median_cutoff.csv.gz
wildcards: genome=hg19, tissue=Brain_Cortex
resources: tmpdir=/tmp
Downloading SpliceMaps...
0%| | 0/98 [00:00<?, ?it/s]../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi5_method=kn_event_filter=median_cutoff.csv.gz
12%|█████████████████▏ | 12/98 [00:01<00:08, 10.70it/s]../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi3_method=kn_event_filter=median_cutoff.csv.gz
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 98/98 [00:02<00:00, 48.24it/s]
[Fri May 24 11:08:07 2024]
Finished job 2.
1 of 7 steps (14%) done
Select jobs to execute...
[Fri May 24 11:08:07 2024]
rule download_human_fasta:
output: ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa
jobid: 1
reason: Missing output files: ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa
resources: tmpdir=/tmp
--2024-05-24 11:08:07-- https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_39/GRCh37_mapping/GRCh37.primary_assembly.genome.fa.gz
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... 193.62.193.165
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.165|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 869924494 (830M) [application/x-gzip]
Saving to: ‘STDOUT’
- 39%[===================================> ] 331.22M 1.54MB/s eta 5m 20s ^- 39%[===================================> ] 331.26M 1.55MB/s in 3m 33s
2024-05-24 12:03:15 (1.56 MB/s) - Connection closed at byte 347351689. Retrying.
--2024-05-24 12:17:52-- (try: 2) https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_39/GRCh37_mapping/GRCh37.primary_assembly.genome.fa.gz
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.165|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 869924494 (830M), 522572805 (498M) remaining [application/x-gzip]
Saving to: ‘STDOUT’
- 100%[++++++++++++++++++++++++++++++++++++=======================================================>] 829.62M 1.80MB/s in 5m 4s
2024-05-24 12:22:56 (1.64 MB/s) - written to stdout [869924494/869924494]
[Fri May 24 12:22:56 2024]
Finished job 1.
2 of 7 steps (29%) done
Select jobs to execute...
[Fri May 24 12:22:57 2024]
rule mmsplice_splicemap:
input: ../data/resources/analysis_files/vcf_files/example_hg19.vcf.gz, ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa, ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi5_method=kn_event_filter=median_cutoff.csv.gz, ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi3_method=kn_event_filter=median_cutoff.csv.gz
output: ../data/results/hg19/model_scores_from_absplice_features/example_hg19.vcf.gz_MMSplice_SpliceMap.csv
jobid: 4
reason: Missing output files: ../data/results/hg19/model_scores_from_absplice_features/example_hg19.vcf.gz_MMSplice_SpliceMap.csv; Input files updated by another job: ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa, ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi3_method=kn_event_filter=median_cutoff.csv.gz, ../data/resources/downloaded_files/splicemap_hg19/Brain_Cortex_splicemap_psi5_method=kn_event_filter=median_cutoff.csv.gz
wildcards: genome=hg19, vcf_id=example_hg19.vcf.gz
resources: tmpdir=/tmp, mem_mb=32000, mem_mib=30518, threads=4
2024-05-24 12:23:00.140524: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
285750it [00:00, 389305.07it/s]
284409it [00:00, 427808.50it/s]
2024-05-24 12:23:36.082493: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0it [00:00, ?it/s]2024-05-24 12:23:37.007840: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:118] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7fffb09f53a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
1it [00:01, 1.26s/it]
[Fri May 24 12:23:38 2024]
Finished job 4.
3 of 7 steps (43%) done
Select jobs to execute...
[Fri May 24 12:23:38 2024]
rule spliceai:
input: ../data/resources/analysis_files/vcf_files/example_hg19.vcf.gz, ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa
output: ../data/results/hg19/model_scores_from_absplice_features/example_hg19.vcf.gz_SpliceAI.vcf
jobid: 6
reason: Missing output files: ../data/results/hg19/model_scores_from_absplice_features/example_hg19.vcf.gz_SpliceAI.vcf; Input files updated by another job: ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa
wildcards: genome=hg19, vcf_id=example_hg19.vcf.gz
resources: tmpdir=/tmp, mem_mb=16000, mem_mib=15259, threads=1, gpu=1
Activating conda environment: .snakemake/conda/9241678ce9daadc3d4184a0af19626a0_
/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py:585: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
np.object,
Traceback (most recent call last):
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/bin/spliceai", line 7, in <module>
from spliceai.__main__ import main
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/spliceai/__main__.py", line 5, in <module>
from spliceai.utils import Annotator, get_delta_scores
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/spliceai/utils.py", line 6, in <module>
from keras.models import load_model
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/keras/__init__.py", line 21, in <module>
from tensorflow.python import tf2
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/__init__.py", line 41, in <module>
from tensorflow.python.tools import module_util as _module_util
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/__init__.py", line 46, in <module>
from tensorflow.python import data
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/__init__.py", line 25, in <module>
from tensorflow.python.data import experimental
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/experimental/__init__.py", line 97, in <module>
from tensorflow.python.data.experimental import service
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/experimental/service/__init__.py", line 353, in <module>
from tensorflow.python.data.experimental.ops.data_service_ops import distribute
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/data_service_ops.py", line 26, in <module>
from tensorflow.python.data.experimental.ops import compression_ops
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/compression_ops.py", line 20, in <module>
from tensorflow.python.data.util import structure
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/util/structure.py", line 26, in <module>
from tensorflow.python.data.util import nest
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/data/util/nest.py", line 40, in <module>
from tensorflow.python.framework import sparse_tensor as _sparse_tensor
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/framework/sparse_tensor.py", line 28, in <module>
from tensorflow.python.framework import constant_op
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py", line 29, in <module>
from tensorflow.python.eager import execute
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 27, in <module>
from tensorflow.python.framework import dtypes
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py", line 585, in <module>
np.object,
File "/app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_/lib/python3.9/site-packages/numpy/__init__.py", line 324, in __getattr__
raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`. To avoid this error in existing code, use `object` by itself. Doing this will not modify any behavior and is safe.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
[Fri May 24 12:23:40 2024]
Error in rule spliceai:
jobid: 6
input: ../data/resources/analysis_files/vcf_files/example_hg19.vcf.gz, ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa
output: ../data/results/hg19/model_scores_from_absplice_features/example_hg19.vcf.gz_SpliceAI.vcf
conda-env: /app/absplice/example/workflow/.snakemake/conda/9241678ce9daadc3d4184a0af19626a0_
shell:
spliceai -I ../data/resources/analysis_files/vcf_files/example_hg19.vcf.gz -O ../data/results/hg19/model_scores_from_absplice_features/example_hg19.vcf.gz_SpliceAI.vcf -R ../data/resources/downloaded_files/GRCh37.primary_assembly.genome.fa -A grch37
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-05-24T110424.018017.snakemake.log
neverov-am
Hey guys, thx very much for your tool!
I just wanted to try it out, because it seems really interesting. However, I stumbled over the following error when I tried to execute the example workflow:
AttributeError: module 'numpy' has no attribute 'object'.Here the whole output:
My set-up: I used the set-up optoin where one uses a Conda-Environment.
My Conda-Env: