Failed to start from a particular task if last update failed

[horkko]

Hi,

I've discovered a little bug when I try to start a postprocess for a bank which failed its last update. Let's say I have BLOCKS=A,B,C. BLOCKS A succeeded but BLOCK B failed. Then the bank update failed and returned an error. If I then decide to rebuild the BLOCK B like: biomaj-cli.py --update --bank test --from-task postprocess --release x --task B it fails saying: 'C' (a KeyError). I've dig into the code and I located the line which throw this exception. It is in session.py in method reload_postprocess_in_order line 97.

for process in processes:
    copy_postprocess[block][meta][process] = postprocess[block][meta][process]

Here postprocess does not have key (block) C because it is not found in MongoDB session.postprocess. Could you have a look at it please?

Thanks

Emmanuel

[osallou]

I gonna have a look, but why asking to restart from fails task. With the workflow, it will restart on failed task.

[horkko]

I wanted to avoid to check again remote data (huge bank embl_wgs) so as I knew download was ok and first BLOCKS, I wanted to start directly from failed BLOCK. But in my point of view, it should not failed no? It should behave the same as restarting an update no?

[osallou]

I think I found a patch (seems to work for me), would you mind testing it before I merge in develop? Branch issue_52

[horkko]

On branch develop it fails as expected:

(biomaj3) biomaj@biomaj [02:27:39] biomaj (develop) $ biomaj-cli.py --update --bank refseq_update --from-task postprocess --release 2016-06-08 --process GOLDEN
2016-06-09 14:27:49,407 INFO  [root][MainThread] [bank.py:__init__:63] Log file: /pasteur/services/banques/biomaj3/log/refseq_update/1465475269.4/refseq_update.log
Log file: /pasteur/services/banques/biomaj3/log/refseq_update/1465475269.4/refseq_update.log
2016-06-09 14:27:49,451 WARNI [root][MainThread] [bank.py:update:999] Bank:refseq_update:Update
2016-06-09 14:27:49,453 INFO  [root][MainThread] [bank.py:update:1009] Bank:refseq_update:Release:2016-06-08
2016-06-09 14:27:49,467 DEBUG [root][MainThread] [bank.py:load_session:746] Load specified session 1465431307.32
'VGOLDEN'

On branch issue_52 it seems to work, at least it does not throw ;)

2016-06-09 14:28:48,634 INFO  [root][Thread-5] [metaprocess.py:run:171] PROC:META:SKIP:PROCESS:blast2_refseqpall
2016-06-09 14:28:48,635 DEBUG [root][MainThread] [processfactory.py:run_threads:80] Meta threads are over
2016-06-09 14:28:48,635 INFO  [root][MainThread] [processfactory.py:run:218] PROC:POST:BLOCK:GOLDEN
2016-06-09 14:28:48,636 DEBUG [root][MainThread] [processfactory.py:run_threads:41] Start meta threads
2016-06-09 14:28:48,655 INFO  [root][Thread-7] [metaprocess.py:run:161] PROC:META:RUN:GOLDEN_REF
2016-06-09 14:28:48,663 INFO  [root][Thread-7] [metaprocess.py:run:174] PROC:META:RUN:PROCESS:golden_refseqp
2016-06-09 14:28:48,667 DEBUG [root][Thread-7] [metaprocess.py:set_progress:151] Process:progress:GOLDEN_REF_golden_refseqp=None
2016-06-09 14:28:48,682 DEBUG [root][Thread-7] [process.py:run:71] PROCESS:EXEC:[u'ngolden.sh', u'refseqpnew_golden.lst', u'/pasteur/services/banques/prod/rel/refseq_update/future_release/golden/3.0', u'refseqp_update', u'refseq_update', u'/pasteur/services/banques/prod/rel/refseq_update/future_release/uncompressed', u'rsnc.*.gpff']
2016-06-09 14:28:48,682 INFO  [root][Thread-7] [process.py:run:74] PROCESS:RUN:GOLDEN_REF_golden_refseqp
...

[osallou]

should I merge or do you want more testing?

[horkko]

Let's wait until the end of the update process? I'll let you know

[horkko]

Ok it terminated ok