Open moldach opened 3 years ago
Are you using a virtualenv? version conflicts happen quite often when using the system install. I would edit the following to
$ export MAVIS_ALIGNER='bwa mem' $ export MAVIS_ALIGNER_REFERENCE=~/MAVIS/reference_inputs/hg19.fa $ git clone https://github.com/bcgsc/mavis.git $ cd mavis $ python3 -m venv venv $ source venv/bin/activate $ pip install zc.buildout $ python bootstrap.py $ bin/buildout
trying this on my system I did not encounter any issues
(venv) [creisle04: mavis]$ python3 -m venv venv
(venv) [creisle04: mavis]$ source venv/bin/activate
(venv) [creisle04: mavis]$ pip install zc.buildout
Looking in indexes: https://pypi.bcgsc.ca/gsc/packages/
Collecting zc.buildout
Downloading https://pypi.bcgsc.ca/root/pypi/+f/088/0f131cd5f3c5f/zc.buildout-2.13.3-py2.py3-none-any.whl (153kB)
100% |████████████████████████████████| 163kB 36.5MB/s
Requirement already satisfied: setuptools>=8.0 in ./venv/lib/python3.7/site-packages (from zc.buildout) (40.6.2)
Installing collected packages: zc.buildout
Successfully installed zc.buildout-2.13.3
You are using pip version 18.1, however version 20.3b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(venv) [creisle04: mavis]$ python bootstrap.py
ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check https://github.com/pypa/setuptools/issues/581 for more info; use pip to install setuptools
Creating directory '/projects/dat/workspace/creisle/tmp/mavis/eggs'.
Creating directory '/projects/dat/workspace/creisle/tmp/mavis/bin'.
Creating directory '/projects/dat/workspace/creisle/tmp/mavis/parts'.
Creating directory '/projects/dat/workspace/creisle/tmp/mavis/develop-eggs'.
Generated script '/projects/dat/workspace/creisle/tmp/mavis/bin/buildout'.
(venv) [creisle04: mavis]$ /projects/dat/workspace/creisle/tmp/mavis/bin/buildout
Develop: '/projects/dat/workspace/creisle/tmp/mavis/.'
warning: no files found matching '*.pl' under directory 'tools'
no previously-included directories found matching 'docs/build'
no previously-included directories found matching 'docs/source/auto'
WARNING: Aligner is required. Missing executable: blat
Getting distribution for 'zc.recipe.egg>=2.0.6'.
Got zc.recipe.egg 2.0.7.
Installing mavis.
Getting distribution for 'svgwrite'.
Got svgwrite 1.4.
Getting distribution for 'shortuuid>=0.5.0'.
Got shortuuid 1.0.1.
Getting distribution for 'pyvcf==0.6.8'.
zip_safe flag not set; analyzing archive contents...
vcf.test.__pycache__.test_vcf.cpython-37: module references __file__
Got PyVCF 0.6.8.
Getting distribution for 'pysam<=0.15.2,>=0.9'.
Got pysam 0.15.2.
Getting distribution for 'numpy>=1.13.1'.
Got numpy 1.19.4.
Getting distribution for 'networkx==1.11.0'.
Got networkx 1.11.
Getting distribution for 'colour'.
Got colour 0.1.5.
Getting distribution for 'braceexpand==0.1.2'.
zip_safe flag not set; analyzing archive contents...
Got braceexpand 0.1.2.
Getting distribution for 'biopython<1.78,>=1.70'.
Got biopython 1.77.
Getting distribution for 'Shapely>=1.6.4.post1'.
Got Shapely 1.7.1.
Getting distribution for 'Distance>=0.1.3'.
notice: no C support available
Warning: 'classifiers' should be a list, got type 'tuple'
zip_safe flag not set; analyzing archive contents...
Got Distance 0.1.3.
Getting distribution for 'decorator>=3.4.0'.
Got decorator 4.4.2.
Generated script '/projects/dat/workspace/creisle/tmp/mavis/bin/calculate_ref_alt_counts'.
Generated script '/projects/dat/workspace/creisle/tmp/mavis/bin/mavis'.
(venv) [creisle04: mavis]$ /projects/dat/workspace/creisle/tmp/mavis/bin/mavis
usage: mavis [-h] [-v]
{annotate,validate,setup,schedule,cluster,pairing,summary,config,convert,overlay}
...
mavis: error: the following arguments are required: command
I do see the sequence error, it results when you don't have mavis on the path so it doesn't know how to build the subcommands. you can fix it by adding mavis to the path first
(venv) [creisle04: mavis]$ export PATH=/projects/dat/workspace/creisle/tmp/mavis/bin:$PATH
(venv) [creisle04: mavis]$ ./bin/mavis setup tests/data/pipeline_config.cfg -o output_dir
MAVIS: 2.2.8
hostname: creisle04.phage.bcgsc.ca
[2020-11-20 09:59:36] arguments
command = 'setup'
config = '/projects/dat/workspace/creisle/tmp/mavis/tests/data/pipeline_config.cfg'
log = None
log_level = 'INFO'
output = 'output_dir'
skip_stage = []
creating output directory: 'output_dir/converted_inputs'
setting up the directory structure for mock-A36971 as /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/cluster'
[2020-11-20 09:59:36] clustering
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/cluster/MC_mock-A36971_batch-65sqBUfUCy6HXUrsuzXAo6.log
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/validate'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/validate/batch-65sqBUfUCy6HXUrsuzXAo6-1'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/validate/batch-65sqBUfUCy6HXUrsuzXAo6-2'
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/validate/submit.sh
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/annotate'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/annotate/batch-65sqBUfUCy6HXUrsuzXAo6-1'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/annotate/batch-65sqBUfUCy6HXUrsuzXAo6-2'
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A36971_diseased_genome/annotate/submit.sh
setting up the directory structure for mock-A47933 as /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/cluster'
[2020-11-20 09:59:36] clustering
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/cluster/MC_mock-A47933_batch-65sqBUfUCy6HXUrsuzXAo6.log
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/validate'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/validate/batch-65sqBUfUCy6HXUrsuzXAo6-1'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/validate/batch-65sqBUfUCy6HXUrsuzXAo6-2'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/validate/batch-65sqBUfUCy6HXUrsuzXAo6-3'
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/validate/submit.sh
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/annotate'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/annotate/batch-65sqBUfUCy6HXUrsuzXAo6-1'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/annotate/batch-65sqBUfUCy6HXUrsuzXAo6-2'
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/annotate/batch-65sqBUfUCy6HXUrsuzXAo6-3'
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/mock-A47933_diseased_transcriptome/annotate/submit.sh
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/pairing'
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/pairing/submit.sh
creating output directory: '/projects/dat/workspace/creisle/tmp/mavis/output_dir/summary'
[2020-11-20 09:59:36] writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/summary/submit.sh
writing: /projects/dat/workspace/creisle/tmp/mavis/output_dir/build.cfg
run time (hh/mm/ss): 0:00:00
run time (s): 0
I will add the documentation flag here tho b/c I think this could definitely have a more helpful error message
Hi I was using a venv
but I think the PATH
was only one issue here.
$ export MAVIS_ALIGNER='bwa mem'
$ export MAVIS_ALIGNER_REFERENCE=~/MAVIS/reference_inputs/hg19.fa
$ git clone https://github.com/bcgsc/mavis.git
$ cd mavis
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install zc.buildout
$ python bootstrap.py
$ bin/buildout
I will start an interactive session with salloc
, source venv
, export PATH
and try to run the example:
$ salloc --time=1:0:0 --mem=12000
$ export PATH=~/bin/mavis/bin:$PATH
$ export MAVIS_ALIGNER='bwa mem'
$ export MAVIS_ALIGNER_REFERENCE=~/MAVIS/reference_inputs/hg19.fa
$ export MAVIS_SCHEDULER=SLURM
$ export MAVIS_CONCURRENCY_LIMIT=2
$ ./bin/mavis setup tests/data/pipeline_config.cfg -o output_dir
$ ./bin/mavis/schedule -o output_dir
$ ./bin/mavis schedule -o output_dir --submit
After a while I check to see if they ran and I see failures.
Looking in one of the logs I see the failure is caused due to blat
:
Traceback (most recent call last):
File "/home/moldach/bin/mavis/bin/mavis", line 24, in <module>
sys.exit(mavis.main.main())
File "/home/moldach/bin/mavis/mavis/main.py", line 465, in main
args.aligner_version = get_aligner_version(args.aligner)
File "/home/moldach/bin/mavis/mavis/align.py", line 174, in get_aligner_version
raise ValueError("unable to parse blat version number from:'{}'".format(proc))
ValueError: unable to parse blat version number from:'/bin/sh: blat: command not found'
This is odd as export MAVIS_ALIGNER='bwa mem'
was used before creating the venv
and the installation?
The test pipeline_config.cfg has the aligner set to blat so mavis won't use the environment variable in this case.
Can you try it again using blat as the aligner?
This is related to #232 I was getting on my laptop - the
"Bio.Alphabet has been removed from Biopython
error. You had suggested trying version2.2.8
so I'm pulling the latest master from github to try.I got this same error on an HPC; however, I got further on the buildout installation here so I'm opening this ticket (to be clear this is a different HPC from that we are trouble-shooting in #228 & #229).
Install using buildout method
Try MAVIS
Seems to be an issue with
networkx==1.11.0
so let's install that.Install
Run again