GRENEPIPE v12.1

[mrese001]

Hi Lucas!

In an effort to troubleshoot as much as I can with Grenepipe I downloaded the latest version on a coworker's directory and am attempting to run a citrus genome through it. When I do, the errors seem to reference MarkDuplicates-java-opts. I read previous issues regarding this and attempted to apply a condition " -Xmx10g" and even scaled up to 40g but with no luck. Any advice on this?

Here is the Error output: Full Traceback (most recent call last): File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/init.py", line 593, in snakemake snakefile, overwrite_first_rule=True, print_compilation=print_compilation File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/workflow.py", line 1114, in include exec(compile(code, snakefile, "exec"), self.globals) File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/Snakefile", line 67, in File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/workflow.py", line 1114, in include exec(compile(code, snakefile, "exec"), self.globals) File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/mapping.smk", line 266, in if config["settings"]["clip-read-overlaps"]: File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/workflow.py", line 1114, in include exec(compile(code, snakefile, "exec"), self.globals) File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/duplicates-picard.smk", line 44, in wrapper: KeyError: 'MarkDuplicates-java-opts'

KeyError in line 35 of /bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/duplicates-picard.smk: 'MarkDuplicates-java-opts' File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/Snakefile", line 57, in File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/mapping.smk", line 211, in File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/duplicates-picard.smk", line 35, in

[lczech]

Hey Mariano,

it seems that you are using a newer version of grenepipe with an older config.yaml - that won't work. It's a bit unfortunate, but while we are in grenedalf pre-v1.0.0, to keep the code simple, releases are not always backwards compatible. So, to solve this, you'll have to use the config.yaml of the version you are using, and adapt that to your dataset. Usually, you can re-use most of it - use the diff or sdiff command on your terminal to find the differences between the two config files, and make changes as to the new one as needed.

Cheers Lucas

[mrese001]

Hey Lucas!

Thanks for the advice. I changed the yaml file to the correct version and now onto what I think may be the source of my stress around this workflow: I found that by typing:

$ conda info

I get the following collapsible details and error. I think this may be because either/both:

1. I have my conda environment in a space with too little memory
2. I am asking snakemake to use miniconda3 from the cluster and not the one locally

Would this be possible? If so how can I ask snakemake to use the local version?

Thanks again and talk soon :)

Details

``` active environment : base active env location : /opt/linux/rhel/8.x/x86_64/pkgs/miniconda3/py39_4.12.0 shell level : 1 user config file : /rhome/rpisc002/.condarc populated config files : conda version : 23.3.1 conda-build version : not installed python version : 3.9.15.final.0 virtual packages : __archspec=1=x86_64 __glibc=2.28=0 __linux=4.18.0=0 __unix=0=0 base environment : /opt/linux/rhel/8.x/x86_64/pkgs/miniconda3/py39_4.12.0 (read only) conda av data dir : /opt/linux/rhel/8.x/x86_64/pkgs/miniconda3/py39_4.12.0/etc/conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/linux/rhel/8.x/x86_64/pkgs/miniconda3/py39_4.12.0/pkgs /rhome/rpisc002/.conda/pkgs envs directories : /rhome/rpisc002/.conda/envs /opt/linux/rhel/8.x/x86_64/pkgs/miniconda3/py39_4.12.0/envs platform : linux-64 user-agent : conda/23.3.1 requests/2.28.1 CPython/3.9.15 Linux/4.18.0-348.12.2.el8_5.x86_64 rocky/8.5 glibc/2.28 UID:GID : 3907:1247 netrc file : None offline mode : False Here is my complete error: Touching output file called/all.scaffold_576.g.done. Full Traceback (most recent call last): File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2347, in run_wrapper edit_notebook, File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/qc-bam.smk", line 311, in __rule_picard_collectmultiplemetrics File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/script.py", line 938, in script executor.evaluate() File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/script.py", line 313, in evaluate self.execute_script(fd.name, edit=edit) File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/script.py", line 506, in execute_script self._execute_cmd("{py_exec} {fname:q}", py_exec=py_exec, fname=fname) File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/script.py", line 354, in _execute_cmd **kwargs File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/shell.py", line 231, in __new__ raise sp.CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'set -euo pipefail; /rhome/rpisc002/.conda/envs/grenepipe/bin/python3.7 /bigdata/koeniglab/rpisc002/GRENEPIPE/grenep ipeWD/parents_draft/.snakemake/scripts/tmpdupx9ezb.picard-collectmultiplemetrics.py' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 568, in _callback raise ex File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run run_func(*args) File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2359, in run_wrapper ex, lineno, linemaps=linemaps, snakefile=file, show_traceback=True snakemake.exceptions.RuleException: CalledProcessError in line 205 of /bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/qc-bam.smk: Command 'set -euo pipefail; /rhome/rpisc002/.conda/envs/grenepipe/bin/python3.7 /bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/parents_draft/.snakemake/ scripts/tmpdupx9ezb.picard-collectmultiplemetrics.py' returned non-zero exit status 1. File "/rhome/rpisc002/.conda/envs/grenepipe/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2347, in run_wrapper File "/bigdata/koeniglab/rpisc002/GRENEPIPE/grenepipeWD/rules/qc-bam.smk", line 205, in __rule_picard_collectmultiplemetrics ```

[lczech]

Hey @mrese001,

yes, those errors look like they could come from mispatching conda/python/pandas versions. You seem to be operating in base environment, and not in a conda environment for grenepipe, as described here. That might be the issue. The beginning of the grenepipe output log lists all the base conda and python versions and environments - can you compare if those match your expectations, and/or post that first bit of the log file here as well? For example, above, it says that python is in version 3.9.15, while grenpipe uses python 3.7.10, so something is not quite right there.

I would highly recommend to follow the setup as describe here, as otherwise you can easily run into such incompatibilities. Please check that this is the case.

Cheers Lucas

[lczech]

Hi @mrese001,

any updates on this?

Cheers Lucas

[mrese001]

Hey @lczech ,

You were indeed correct - I had the base environment loaded instead of the build for grenepipe. Sorry it took so long for me to respond to your replies!

Mariano