snakemake.exceptions.WorkflowError: Config file is not valid JSON or YAML

[mattdoug604]

Hello, I am trying to run MAVIS and I am getting error. -bash-4.2$ snakemake -j 1 --configfile mavis.cfg -s Snakefile Traceback (most recent call last): File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/io.py", line 1604, in _load_configfile return yaml.load(f, Loader=OrderedLoader) File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/init.py", line 114, in load return loader.get_single_data() File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/composer.py", line 39, in get_single_node if not self.check_event(StreamEndEvent): File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/parser.py", line 171, in parse_document_start raise ParserError(None, None, yaml.parser.ParserError: expected '', but found '' in "mavis.cfg", line 2, column 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/u/home/m/mdistler/project-jflint/anaconda3/bin/snakemake", line 33, in sys.exit(load_entry_point('snakemake==6.15.5', 'console_scripts', 'snakemake')()) File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/init.py", line 2757, in main success = snakemake( File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/init.py", line 500, in snakemake update_config(overwrite_config, load_configfile(f)) File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/io.py", line 1617, in load_configfile config = _load_configfile(configpath) File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/io.py", line 1606, in _load_configfile raise WorkflowError( snakemake.exceptions.WorkflowError: Config file is not valid JSON or YAML. In case of YAML, make sure to not mix whitespace and tab indentation.

Thank you for help

[543090lee]

[metadata] name = mavis version = 3.0.0 url = https://github.com/bcgsc/mavis.git download_url = https://github.com/bcgsc/mavis/archive/v2.2.10.tar.gz description = A Structural Variant Post-Processing Package author_email = creisle@bcgsc.ca author = Caralyn Reisle maintainer_email = mavis@bcgsc.ca maintainer = mavis long_description = file: README.md long_description_content_type = text/markdown license_file = LICENSE project_urls = mavis = http://mavis.bcgsc.ca

[bdist_wheel] universal = 1

[pycodestyle] ignore = E501 W503 E203 statistics = True

[flake8] ignore = E501,W503,E203

[options] packages = find: package_dir = = src python_requires = >=3.7 dependency_links = [] include_package_data = True install_requires = biopython>=1.70, <1.78 braceexpand==0.1.2 colour Distance>=0.1.3 mavis_config>=1.1.0, <2.0.0 networkx>=2.5,<3 numpy>=1.13.1 pandas>=1.1, <2 pysam Shapely>=1.6.4.post1 shortuuid>=0.5.0 svgwrite typing_extensions>=4 setup_requires = pip>=9.0.0 setuptools>=36.0.0

[options.packages.find] exclude = tests where = src

[options.extras_require] doc = mkdocs>=1.1.2 markdown-refdocs mkdocs-material>=5.4.0 markdown-include mkdocs-simple-hooks>=0.1.2 mkdocs-awesome-pages-plugin test = timeout-decorator>=0.3.3 coverage>=4.2 pycodestyle>=2.3.1 pytest pytest-cov dev = black flake8 twine wheel timeout-decorator>=0.3.3 coverage>=4.2 pycodestyle>=2.3.1 pytest pytest-cov pytest-xdist mkdocs>=1.1.2,<2 markdown-refdocs mkdocs-material>=5.4.0 markdown-include mkdocs-simple-hooks>=0.1.2 types-setuptools>=57.4.7, <58 deploy = twine wheel tools = pyensembl simplejson requests

[options.entry_points] console_scripts = mavis = mavis.main:main calculate_ref_alt_counts = tools.calculate_ref_alt_counts:main

[zhemingfan]

Hi @543090lee and @mattdoug604, the files you posted are not MAVIS config files.

MAVIS configuration files are used to specify the specific parameters you want to use for each step of the MAVIS pipeline. This file needs to be a json. An example of a config can be found at mini-tutorial.config.json. The full list of customizable settings can be found in our separate mavis_config repository.

[543090lee]

I tried running the mini-tutorial.config.json and I followed all the instructions on mavis website, and got error below. (base) [seichang@login3 project-zarlab]$ snakemake -j 1 --configfile=tests/mini-tutorial.config.json -s Snakefile Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Singularity containers: ignored Traceback (most recent call last): File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/init.py", line 785, in snakemake keepmetadata=keep_metadata, File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/workflow.py", line 1060, in execute logger.run_info("\n".join(dag.stats())) File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/dag.py", line 2191, in stats yield tabulate(rows, headers="keys") File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 2049, in tabulate tabular_data, headers, showindex=showindex File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 1471, in _normalize_tabular_data rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows)) File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 1471, in rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows)) File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 107, in _is_separating_line (len(row) >= 1 and row[0] == SEPARATING_LINE) File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/rules.py", line 1138, in eq return self.name == other.name and self.output == other.output AttributeError: 'str' object has no attribute 'name'

Thank you for your help

[zhemingfan]

Hi @543090lee, thank you for the question. The issue stems from the tabulate dependency included in Snakemake. We currently have a PR that's targeted to fix this. Once its merged, could you re-install and try it again?

[543090lee]

Hello, Thank you for the comment. Are there any updates on the PR that's targeted to fix the issue stemming from the tabulate dependency issue?

Thank you for your help.

[zhemingfan]

Hi @543090lee, try out the latest develop_v3 branch. It should be working now after the merge from the latest PR https://github.com/bcgsc/mavis/pull/337

[ziphra]

I had the same problem than @543090lee, I installed the latest develop_v3 branch with git clone -b develop_v3 https://github.com/bcgsc/mavis.git but it didnt work.

However, after reinstalling with pip and doing pip3 install tabulate==0.8.9 the mini tutorial worked

[543090lee]

Thank you for your help @ziphra and @zhemingfan I followed the above instruction, and I run the mini tutorial and still got the following error. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Singularity containers: ignored Job stats: job count min threads max threads

---

all 1 1 1 annotate 7 1 1 cluster 2 1 1 convert 1 1 1 copy_config 1 1 1 init_config 1 1 1 pairing 1 1 1 summary 1 1 1 validate 7 1 1 total 22 1 1

Select jobs to execute...

[Thu Nov 3 16:26:44 2022] rule copy_config: output: output_dir/config.raw.json log: output_dir/logs/copy_config.snakemake.log.txt jobid: 8 resources: tmpdir=/tmp, time_limit=57600, mem_mb=4000, cpus=1, log_dir=output_dir/logs

[Thu Nov 3 16:26:46 2022] Finished job 8. 1 of 22 steps (5%) done Select jobs to execute...

[Thu Nov 3 16:26:46 2022] rule init_config: input: output_dir/config.raw.json output: output_dir/config.json log: output_dir/logs/init_config.snakemake.log.txt jobid: 7 resources: tmpdir=/tmp, time_limit=57600, mem_mb=16000, cpus=1, log_dir=output_dir/logs

/usr/bin/bash: mavis: command not found [Thu Nov 3 16:26:46 2022] Error in rule init_config: jobid: 7 output: output_dir/config.json log: output_dir/logs/init_config.snakemake.log.txt (check log file(s) for error message) shell: mavis setup --config output_dir/config.raw.json --outputfile output_dir/config.json (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /u/project/zarlab/seichang/mavis/.snakemake/log/2022-11-03T162643.244861.snakemake.log

Thank you

[creisle]

Singularity containers: ignored

@543090lee from your log above it looks like you are not using the container for the workflow is that correct?

Because of this, the workflow is looking for mavis to be available on the command line which is why you are seeing the error below

/usr/bin/bash: mavis: command not found

We should have added that to the documentation. I will make an update to the docs. You have 2 options.

1. add the --use-singularity flag to the last command in the tutorial
2. install mavis via pip as well as the aligners (see user manual) to be available on the default PATH

[creisle]

I had the same problem than @543090lee, I installed the latest develop_v3 branch with git clone -b develop_v3 https://github.com/bcgsc/mavis.git but it didnt work.

However, after reinstalling with pip and doing pip3 install tabulate==0.8.9 the mini tutorial worked

Release v3.1.0 should fix the tabulate error

[543090lee]

Hello, I installed the new version 3.1.0, and ran the mini tutorial and got this error. (base) [seichang@login4 mavis]$ snakemake -j 1 --configfile=tests/mini-tutorial.config.json -s Snakefile Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Singularity containers: ignored Job stats: job count min threads max threads

---

all 1 1 1 annotate 7 1 1 cluster 2 1 1 convert 1 1 1 copy_config 1 1 1 init_config 1 1 1 pairing 1 1 1 summary 1 1 1 validate 7 1 1 total 22 1 1

Select jobs to execute...

[Mon Nov 28 16:23:43 2022] rule copy_config: output: output_dir/config.raw.json log: output_dir/logs/copy_config.snakemake.log.txt jobid: 8 resources: tmpdir=/tmp, time_limit=57600, mem_mb=4000, cpus=1, log_dir=output_dir/logs

[Mon Nov 28 16:23:44 2022] Finished job 8. 1 of 22 steps (5%) done Select jobs to execute...

[Mon Nov 28 16:23:44 2022] rule init_config: input: output_dir/config.raw.json output: output_dir/config.json log: output_dir/logs/init_config.snakemake.log.txt jobid: 7 resources: tmpdir=/tmp, time_limit=57600, mem_mb=16000, cpus=1, log_dir=output_dir/logs

/usr/bin/bash: mavis: command not found [Mon Nov 28 16:23:44 2022] Error in rule init_config: jobid: 7 output: output_dir/config.json log: output_dir/logs/init_config.snakemake.log.txt (check log file(s) for error message) shell: mavis setup --config output_dir/config.raw.json --outputfile output_dir/config.json (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /u/project/zarlab/seichang/mavis/.snakemake/log/2022-11-28T162342.743481.snakemake.log

Thank you

[zhemingfan]

Hi @543090lee , what was the error in the log?