Batch module updates - Summer 2024

[edmundmiller]

[!WARNING] Please open PRs against the batch_update_staging branch. Then we can have small, easy to review, atomic PRs but also merge all batch updates to the modules repo in one go.

### Tasks
- [ ] https://github.com/nf-core/modules/issues/4517
- [ ] https://github.com/nf-core/modules/issues/5834
- [ ] https://github.com/nf-core/modules/issues/5832
- [ ] https://github.com/nf-core/modules/issues/5833
- [ ] https://github.com/nf-core/modules/issues/4983
- [ ] https://github.com/nf-core/modules/issues/5830
- [ ] https://github.com/nf-core/modules/issues/5831
- [ ] https://github.com/nf-core/modules/issues/5829
- [ ] https://github.com/nf-core/modules/issues/5835
- [ ] https://github.com/nf-core/modules/issues/5836
- [ ] https://github.com/nf-core/modules/issues/5848

Just a heads up for bulk changes: there is a strict matrix limit of 256 tests in a test strategy on github, so that is the max number of module changes in a PR. @mashehu #2738

[SPPearce]

Can we automatically swap the test datasets paths too?

[ewels]

@SPPearce can you elaborate / open an issue about this please? (Or point to an existing one)

[SPPearce]

@SPPearce can you elaborate / open an issue about this please? (Or point to an existing one)

We have swapped the file paths used in tests from: file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true) to direct paths like: file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)

This is in the guidelines now, but could do with a bulk update.

Made an issue: https://github.com/nf-core/modules/issues/5848

[muffato]

My concern is that several of the changes require Nextflow 24.04. It means that from this point forward, nf-core will only support Nextflow 24.04+. Is there an analysis of 24.04 vs 22.04-23.10 and what developers have to do to make their pipelines compatible with 24.04 ? Any pipeline developer who is unable to move to 24.04 would remain stuck on pre-June 2024 modules and sub-workflows.

[ewels]

@muffato this has been policy for a long time now in nf-core. We are not adopting anything that requires an edge release of Nextflow here, everything is stable. There are other updates that are in 24.04 that we also want to update for pipelines, such as the use of resourceLimits. Some of those changes have been requested for ~6 years, and have been present in the Nextflow edge release for ~6 months.

We will soon bump the minimum required version of Nextflow in the pipeline template, and many / most of these changes should come through automatically via template syncs and module updates. We will do our best to write lint tests to cover most of them so that any required changes don't slip through the gaps. We've also recently started the practice of writing blog posts and doing bytesize talks where we talk over and demonstrate the updates that are required for new versions of the template.

I appreciate that it can be frustrating to keep updating pipelines with these kinds of changes. But that's also part of what makes nf-core great. We are a large pool of feature requests to Nextflow which makes it better. Stability is still possible through using previous versions of Nextflow and pipelines. I don't think that we should be held back from implementing new features once they are available in stable versions of Nextflow.

Any pipeline developer who is unable to move to 24.04

The hope is that no-one falls into this camp. If you can see any potential reasons for this, then please say and we can try to resolve them.

[muffato]

Oh yes @ewels , I don't deny the advantages of moving over. There will be a lot of benefits for people to move. Hopefully we can get away with it by just updating the Nextflow version number and can do pipeline template updates later.

[famosab]

Should this also include adding stubs? (That is not a breaking change but might still be worth to keep in mind?) Related to #4570.