Failed job from Mini Tutorial with no CRASH report in the schedule output

moldach commented 4 years ago

MAVIS version: 2.2.6

Python version: 3.8.0

OS: CentOS Linux release 7.5.1804 (Core)

I am using a shared academic HPC (SLURM scheduler) and do not have root permissions. Therefore, I have tried to install MAVIS and run locally.

I follow along with the steps to the Mini Tutorial but after using the --submit FLAG the jobs show as NOT SUBMITTED and there are missing log file: warnings/errors under the validate section.

Install locally

mkdir /scratch/moldach/TEST; cd /scratch/moldach/TEST
virtualenv myEnv
source myEnv/bin/activate
pip install Cython
pip install mavis

git clone https://github.com/bcgsc/mavis.git
mv mavis/tests .
rm -r mavis
export MAVIS_SCHEDULER=LOCAL
export MAVIS_CONCURRENCY_LIMIT=2
mavis setup tests/data/pipeline_config.cfg -o output_dir
mavis schedule -o output_dir/ --submit

Here is where I see the errors:

mavis schedule -o output_dir/

MAVIS: 2.2.6
                      hostname: gra-login2
[2020-03-17 15:20:14] arguments
                        command = 'schedule'
                        log = None
                        log_level = 'INFO'
                        output = 'output_dir/'
                        resubmit = False
                        submit = False
[2020-03-17 15:20:14] validate
                        MV_mock-A36971_batch-kbpzPF4moQbvaEmVUutgiN-1 (4GQM4c2RWQin8bTPX4zhxe) is UNKNOWN
                          missing log file: /scratch/moldach/TEST/output_dir/mock-A36971_diseased_genome/validate/batch-kbpzPF4moQbvaEmVUutgiN-1/job-MV_mock-A36971_batch-kbpzPF4moQbvaEmVUutgiN-1-4GQM4c2RWQin8bTPX4zhxe.log
                        MV_mock-A36971_batch-kbpzPF4moQbvaEmVUutgiN-2 (bTFwDtnTAtWa5GkfuG9ufm) is UNKNOWN
                          missing log file: /scratch/moldach/TEST/output_dir/mock-A36971_diseased_genome/validate/batch-kbpzPF4moQbvaEmVUutgiN-2/job-MV_mock-A36971_batch-kbpzPF4moQbvaEmVUutgiN-2-bTFwDtnTAtWa5GkfuG9ufm.log
                        MV_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-1 (Kv9YS9Evo9wkFbmJsgVafZ) is UNKNOWN
                          missing log file: /scratch/moldach/TEST/output_dir/mock-A47933_diseased_transcriptome/validate/batch-kbpzPF4moQbvaEmVUutgiN-1/job-MV_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-1-Kv9YS9Evo9wkFbmJsgVafZ.log
                        MV_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-2 (Xj53G6Rfw27edaWsLJ5pH6) is UNKNOWN
                          missing log file: /scratch/moldach/TEST/output_dir/mock-A47933_diseased_transcriptome/validate/batch-kbpzPF4moQbvaEmVUutgiN-2/job-MV_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-2-Xj53G6Rfw27edaWsLJ5pH6.log
                        MV_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-3 (TNrH35hvi5GuasjsVLR8hp) is UNKNOWN
                          missing log file: /scratch/moldach/TEST/output_dir/mock-A47933_diseased_transcriptome/validate/batch-kbpzPF4moQbvaEmVUutgiN-3/job-MV_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-3-TNrH35hvi5GuasjsVLR8hp.log
[2020-03-17 15:20:14] annotate
                        MA_mock-A36971_batch-kbpzPF4moQbvaEmVUutgiN-1 is NOT SUBMITTED
                        MA_mock-A36971_batch-kbpzPF4moQbvaEmVUutgiN-2 is NOT SUBMITTED
                        MA_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-1 is NOT SUBMITTED
                        MA_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-2 is NOT SUBMITTED
                        MA_mock-A47933_batch-kbpzPF4moQbvaEmVUutgiN-3 is NOT SUBMITTED
[2020-03-17 15:20:14] pairing
                        MP_batch-kbpzPF4moQbvaEmVUutgiN is NOT SUBMITTED
[2020-03-17 15:20:14] summary
                        MS_batch-kbpzPF4moQbvaEmVUutgiN is NOT SUBMITTED
                      rewriting: output_dir/build.cfg

When I try to run with the `--resubmit FLAG:

(myEnv) [moldach@gra-login2 TEST]$ mavis schedule -o output_dir/ --resubmit
                      MAVIS: 2.2.6
                      hostname: gra-login2
[2020-03-17 15:35:24] arguments
                        command = 'schedule'
                        log = None
                        log_level = 'INFO'
                        output = 'output_dir/'
                        resubmit = True
                        submit = False
[2020-03-17 15:35:25] validate
                      rewriting: output_dir/build.cfg
Traceback (most recent call last):
  File "/scratch/moldach/TEST/myEnv/bin/mavis", line 8, in <module>
    sys.exit(main())
  File "/scratch/moldach/TEST/myEnv/lib/python3.8/site-packages/mavis/main.py", line 414, in main
    raise err
  File "/scratch/moldach/TEST/myEnv/lib/python3.8/site-packages/mavis/main.py", line 389, in main
    code = pipeline.check_status(log=_util.LOG, **args)
  File "/scratch/moldach/TEST/myEnv/lib/python3.8/site-packages/mavis/schedule/pipeline.py", line 663, in check_status
    run_time = self._job_status(job, submit=submit, resubmit=resubmit, log=log.indent())
  File "/scratch/moldach/TEST/myEnv/lib/python3.8/site-packages/mavis/schedule/pipeline.py", line 604, in _job_status
    self._resubmit_job(job)
  File "/scratch/moldach/TEST/myEnv/lib/python3.8/site-packages/mavis/schedule/pipeline.py", line 541, in _resubmit_job
    self.scheduler.cancel(job)
  File "/scratch/moldach/TEST/myEnv/lib/python3.8/site-packages/mavis/schedule/scheduler.py", line 91, in cancel
    raise NotImplementedError('abstract method')
NotImplementedError: abstract method

I would appreciate your help solving this issue.

creisle commented 4 years ago

Hi @moldach ! thanks for reporting this, we'll have someone look into is ASAP and get back to you

calchoo commented 4 years ago

Hi @moldach ,

Can you see if you have blat installed in the $PATH when you're running the submit? It's a dependency that is needed for the mini tutorial.

You'll have to remake the output_dir

which blat
rm -r output_dir/
mavis setup tests/data/pipeline_config.cfg -o output_dir
mavis schedule -o output_dir/ --submit

moldach commented 4 years ago

This solved the problem, thanks

bcgsc / mavis