CollectMultipleMetrics: there is no package called ‘tidyverse’

[bensprung]

I am not sure what I did that changed things since getting grenepipe to work successfully a few months ago, but I now get an error on step 8 saying that picard-collectmultiplemetrics.py returned non-zero exit status 1. If I look in the picard logs the problem is

INFO    2021-09-29 14:19:26 RExecutor   Executing R script via command: Rscript /tmp/ben/script10615150585100640491.R /home/ben/grenepipe_try_2/qc/picard/121B01-1.base_distribution_by_cycle_metrics /home/ben/grenepipe_try_2/qc/picard/121B01-1.base_distribution_by_cycle.pdf 121B01-1.bam 
ERROR   2021-09-29 14:19:26 ProcessExecutor Error in library(tidyverse) : there is no package called ‘tidyverse’

Confusingly, tidyverse does seem to be installed in the environment (see below) so I am not sure how to try to fix this. I did download the latest grenepipe and also get this error in addition to it occurring with my older version.

(snakemake) ben@OptiPlex990:~/grenepipe-master-2021Sep29$ echo "library(tidyverse)" > test.R
(snakemake) ben@OptiPlex990:~/grenepipe-master-2021Sep29$ Rscript test.R
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
✔ ggplot2 3.3.3     ✔ purrr   0.3.4
✔ tibble  3.1.1     ✔ dplyr   1.0.5
✔ tidyr   1.1.3     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.1

[lczech]

Hi Ben,

thanks for reporting this issue, that is indeed a funny one. I did not know that picard internally calls R scripts. Modern software... conda calling python calling snakemake calling python calling java calling R - no wonder that this is so fragile!

Anyway, should be solved in the latest commit - please pull and try again! If not, feel free to re-open this issue!

Cheers and so long Lucas

PS: I saw that you edited your text quite a bit - and assume that you fixed all the problems that are now gone? In short:

• The error validating the config file is because newer versions of grenepipe tend to include more options, and I always make them required in the config file to make sure that users don't accidentally use outdated config files. That unfortunately means that the config file has to be fixed or re-done for every grenepipe version... but we are pre-1.0, so I feel this is a necessary evil...
• Input files with incompatible contigs: That seems to stem from changed reference files not fitting a particular run any more, or something. Usually, starting from scratch helps.

Let me know if there is anything else that does not work as it should ;-)

[bensprung]

Hi Lucas yeah sorry about those edits, I ran into a couple problems that I was too quick to post about, exactly as you said one was about using the old config file and one was about not deleting all the output of a partially completed run. Thank you for the quick update, I am running now and have passed step 9 so I think it is solved.

[lczech]

Hey Ben,

no need to apologize, that is what the edit feature is for. I was probably just being overly thorough there ;-)

Glad to hear that it works now. NB: The "step number" is more or less meaningless though, as they can be reordered by snakemake, and differ very much depending on your config (i.e., which tools are used) and your data (both the number of samples and the number of reference sequences/contigs affect the total job count). So, a better way to check if a certain thing works now is to look for its rule name (and, if it only fails for certain data, for its input data or wildcards), to make sure that a step that failed earlier now works. Hope that makes sense.

So if there is still more trouble, let me know! Lucas

[bensprung]

Got it, it does. Well, the whole pipeline is working and producing usable output at the moment so I am happy ;)

[lczech]

Awesome! I feel that grenepipe is nearing a 1.0 release. It has been quite the struggle to fix all dependency issues for all users and clusters that it has been used on so far... One would think that the whole idea of conda is to fix those issues once and for all - as this issue (yet again) showcased, this is not true. So, thanks for being a patient beta-tester ;-)

[bensprung]

No prob. I look forward to citing it knock on wood!