[Question] CI for Variant Calling

[Krannich479]

Is your feature request related to a problem? Please describe. No problem here, just a question and an offer.

Describe the solution you'd like I am currently developing a framework for continuous integration and evaluation of small variant calling. I read the GH Actions of poreCov and saw that the current tests are running dry runs testing principal functionality. What I offer is to write a GH Action to benchmark the variant calling (precision/recall/F1-score on synthetic data). Is that generally in your interest or beneficial? Are you already running such tests locally or otherwise outside of GH Actions?

[replikation]

I would prefer to run automatic complete tests via git action, but the question would be, who is financing this ;)? We added test profiles so we run outside GH Action test runs. Important containers are tested directly with real data prior to deployment. So the workflow logic is tested, and some of the important containers are.

Also, since the workflow uses containerization, it would need to download and run multiple containers, which would require more space for the gitaction runs, I think.

But maybe I misunderstood what you mean by benchmark. we could always add such tests to out test profile runs. (listed in the --help https://github.com/replikation/poreCov/blob/ff6cb14e233a328fb31075155626f0b2b58299bf/poreCov.nf#L559-L563)

[Krannich479]

Valid points. When I came up with this idea I totally forgot that poreCov is deploying a number of containers that in total might exceed the GH Actions runner's resources. Also, the most relevant aspect to me is the variant calling which (I just found out) has a quite elaborate test suite within the ARTIC pipeline. I'll close this issue for now.

[hoelzer]

Agree, but maybe the "test profile" Option as mentioned by @replikation could be an interesting use case @Krannich479 . "test_vcf" could download simulated example data and a "gold standard" vcf and then let's porecov produce a new vcf which then can be compared.

Might be neat to test new medaka versions and models.

The test run would then happen outside any GitHub action locally

[MarieLataretu]

Yes, I agree with @hoelzer; it'd still be useful and interesting outside of GH!

We have GH actions using containers for some nextflow pipelines under a (free (?)) organization account - so far, I have not encountered problems with GH actions (e.g. clean and CoVpipe2). But I have to say that I have no overview of the limits, especially with non-organization accounts

[Krannich479]

Okay okay @MarieLataretu @hoelzer @replikation, I'll have to look into how this works with profiles + nxf but I'll give it an attempt.