search

gagneurlab / concise

Concise: Keras extension for regulatory genomics
https://i12g-gagneurweb.in.tum.de/public/docs/concise/
MIT License
32 stars 13 forks source link

Issue raised within concise while running `snakemake` in Manuscript_Avsec_Bioinformatics_2017 #7

Closed LucaCappelletti94 closed 5 years ago

LucaCappelletti94 commented 5 years ago 
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   all
    20  train_exclude_features
    21

[Sat Dec  1 13:52:44 2018]
rule train_exclude_features:
    input: data/eclip/processed/design_matrix/train/TBRG4_extended.csv, data/eclip/processed/design_matrix/valid/TBRG4_extended.csv, data/eclip/processed/design_matrix/test/TBRG4_extended.csv, Scripts/RBP/Eclip/predictive_models/train_exclude_features.py
    output: data/eclip/processed/feature_exclusion_exp/results/TBRG4/DeepNN_scalar_position_ext_gam-excl-polya,gene_end.json
    jobid: 26
    wildcards: rbp_name=TBRG4, exp=DeepNN_scalar_position_ext_gam, fset=polya,gene_end

Using TensorFlow backend.
INFO:2018-12-01 13:52:57,891:excl_f] used_features: ['tss', 'exon_intron', 'intron_exon', 'start_codon', 'stop_codon', 'gene_start']
2018-12-01 13:52:57,891 [INFO] used_features: ['tss', 'exon_intron', 'intron_exon', 'start_codon', 'stop_codon', 'gene_start']
INFO:2018-12-01 13:52:57,891:excl_f] get the best hyper-parameters for a model
2018-12-01 13:52:57,891 [INFO] get the best hyper-parameters for a model
INFO:2018-12-01 13:52:57,892:excl_f] c_exp_name: DeepNN_scalar_position_ext_gam_TBRG4
2018-12-01 13:52:57,892 [INFO] c_exp_name: DeepNN_scalar_position_ext_gam_TBRG4
2018-12-01 13:52:57,892 [INFO] PROTOCOL mongo
2018-12-01 13:52:57,892 [INFO] USERNAME None
2018-12-01 13:52:57,892 [INFO] HOSTNAME localhost
2018-12-01 13:52:57,892 [INFO] PORT 27017
2018-12-01 13:52:57,892 [INFO] PATH /RBP__Eclip/jobs
2018-12-01 13:52:57,892 [INFO] AUTH DB None
2018-12-01 13:52:57,892 [INFO] DB RBP__Eclip
2018-12-01 13:52:57,892 [INFO] COLLECTION jobs
Traceback (most recent call last):
  File "Scripts/RBP/Eclip/predictive_models/train_exclude_features.py", line 75, in <module>
    tid = tr.best_trial_tid()
  File "/home/cappelletti/code/.virtualenvs/virtual-py36gpu/lib/python3.6/site-packages/concise/hyopt.py", line 137, in best_trial_tid
    lid = np.where(np.argsort(losses).argsort() == rank)[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0
[Sat Dec  1 13:52:58 2018]
Error in rule train_exclude_features:
    jobid: 26
    output: data/eclip/processed/feature_exclusion_exp/results/TBRG4/DeepNN_scalar_position_ext_gam-excl-polya,gene_end.json

RuleException:
CalledProcessError in line 123 of /data/Avsec/automated/Manuscript_Avsec_Bioinformatics_2017/Scripts/RBP/Eclip/Snakefile:
Command ' set -euo pipefail;  python Scripts/RBP/Eclip/predictive_models/train_exclude_features.py --rbp=TBRG4 --feature_set=polya,gene_end --exp=DeepNN_scalar_position_ext_gam ' returned non-zero exit status 1.
  File "/data/Avsec/automated/Manuscript_Avsec_Bioinformatics_2017/Scripts/RBP/Eclip/Snakefile", line 123, in __rule_train_exclude_features
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /data/Avsec/automated/Manuscript_Avsec_Bioinformatics_2017/Scripts/RBP/Eclip/.snakemake/log/2018-12-01T135243.477113.snakemake.log
Avsecz commented 5 years ago

I think one would need to add a check if there are no existing experiments tracked and add something like

if losses is not None and len(losses) > 0:
    lid = np.where ....