nvnieuwk
commented
6 months ago Have you tried deleting the cache?
rm -rf ~/.nextflow/plugins/nf-schema/Closed Tobiaspk closed 3 weeks ago
nvnieuwk
commented
6 months ago Have you tried deleting the cache?
rm -rf ~/.nextflow/plugins/nf-schema/
awgymer
commented
6 months ago Confirming I can recreate this error with a brand new install of nf-schema and the above steps.
Not clear exactly what the issue is, given validateParameters imports fine, and the import appears to work in the tests for the plugin.
I guess it may relate to the fact the method is defined with overloading (none of the other methods imported successfully are)
Tobiaspk
commented
6 months ago Have you tried deleting the cache?
rm -rf ~/.nextflow/plugins/nf-schema/
Removing cache did not fix this for me.
Confirming I can recreate this error with a brand new install of
nf-schemaand the above steps.Not clear exactly what the issue is, given
validateParametersimports fine, and the import appears to work in the tests for the plugin.I guess it may relate to the fact the method is defined with overloading (none of the other methods imported successfully are)
Odd, validateParameters, paramsHelp, paramsSummaryMap all worked for me, even after rm cache.
nvnieuwk
commented
6 months ago What version of Nextflow are you using? What operating system are you using? What's the version of nf-core tools you used?
At first I would think this is some nextflow bug, but I'm not sure yet. I'll give it a try later
nvnieuwk
commented
6 months ago NVM, my last post :grin: all the answers are in your first message
nvnieuwk
commented
6 months ago I just tried the steps you laid out with a fresh install of nf-schema. I didn't get any issues with it :/. Can you point me to a repository where you have this issue?
awgymer
commented
6 months ago When you say a fresh install do you mean a manual install from source? Or letting NF pull it?
FWIW my setup used
Nextflow 23.10.1 Mac OS Sonoma (M1)
nvnieuwk
commented
6 months ago I deleted the cache and let Nextflow redownload it.
nvnieuwk
commented
6 months ago @Tobiaspk and @awgymer I managed to recreate the error using Nextflow version 23.10.1. I was using verion 24.02.0-edge in my test. Can you guys confirm on your end that it works on the newer edge releases?
nvnieuwk
commented
6 months ago After some additional testing I also managed to get it to work by converting all include statements to use nf-schema instead of nf-validation. So these should also be updated in subworkflows/nf-core/utils_nfvalidation_plugin and in your main pipeline workflow <pipelinename>.nf
awgymer
commented
6 months ago Ok I have a theory and a bit of testing has me believing I am correct but paging @bentsherman as a groovy/nextflow expert who can hopefully confirm.
All the following can be recreated with nextflow.config:
plugins {
id 'nf-validation@1.1.3' // Validation of pipeline parameters and creation of an input channel from a sample sheet
id 'nf-schema@2.0.0'
}So here goes...
main_schema.nf
include { validateParameters; samplesheetToList } from 'plugin/nf-schema'
workflow {
println "A test"
}Ouput:
A testmain_validation.nf
include { validateParameters; fromSamplesheet } from 'plugin/nf-validation'
workflow {
println "A test"
}Output:
A testmain_schema_validation.nf
include { validateParameters; samplesheetToList } from 'plugin/nf-schema'
include { fromSamplesheet } from 'plugin/nf-validation'
workflow {
println "A test"
}Output:
ERROR ~ Extension 'fromSamplesheet' it isn't defined by plugin nf-validationmain_validation_schema.nf
include { fromSamplesheet } from 'plugin/nf-validation'
include { validateParameters; samplesheetToList } from 'plugin/nf-schema'
workflow {
println "A test"
}Output:
ERROR ~ Extension 'samplesheetToList' it isn't defined by plugin nf-schemaSo the issue... 🥁
Both nf-schema and nf-validation have their actual plugin code under src/main/nextflow/validation and thus in the groovy code as nextflow.validation.
I am guessing that what happens is that both plugins are "loaded" but the compiled code of the first one loaded is what is actually available under both plugins.
This leads to this seemingly impossible successful include (remember samplesheetToList is not in nf-validation):
main_schema_validation_wrong_import.nf
include { validateParameters } from 'plugin/nf-schema'
include { samplesheetToList } from 'plugin/nf-validation'
workflow {
println "A test"
}Output:
ERROR ~ Extension 'samplesheetToList' it isn't defined by plugin nf-schemaThank you for investigating! Does this issue still occur with the edge versions of Nextflow? (just curious :grin:)
awgymer
commented
6 months ago Seems like the issue is not present in 24.01
I am only able to do this surface level testing though so I am not clear what happens for methods that exist in both extensions e.g. validateParameters - whether the correct version for the plugin is installed or if that is yet another possible edge-case!
Will need someone more keyed in with nextflow (and the underlying changes in 24 vs 23 I guess) to explain further.
bentsherman
commented
6 months ago Nextflow was upgraded from Groovy 3 to 4 in 24.01.0-edge, which might have something to do with it. But the basic issue seems to be that both plugins are using the same namespace and have some conflicting extensions? I see they both have a validateParameters function
awgymer
commented
6 months ago Technically the issue isn't the conflicting extensions really (eg validateParamters). It's that they share a "code namespace" (nextflow.validation) whilst appearing to be unique namespaces to the naive user plugin/nf-schema vs plugin/nf-validation.
If nothing in the plugin arch has changed that would support this but the groovy 3->4 change is the cause of the conflict not occurring in edge it makes me even more concerned that it may only be a partial resolution or concealing more edge-cases/bugs?
bentsherman
commented
6 months ago Using the same namespace by itself isn't necessarily a problem, only if you have two classes with the same fully qualified name e.g. nextflow.validation.SchemaValidator.
At the same time, a major version bump like Groovy 3 -> 4 can bring many mysterious changes. If the issue went away in Groovy 4, maybe they improved something in the class loader which can handle the duplicate class names correctly.
nvnieuwk
commented
3 weeks ago Some additional checks have been implemented in nf-core version 3.0 that should prevent this from happening if your pipeline passes the linting checks.
Problem occured during migration from
nf-vallidationtonf-schema. Aim is to migrate to newere sample sheet reader function.Reproducible steps
nextflow.configsubworkflows/local/utils_nfcore_training_pipeline/main.nfError stack