search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.76k stars 629 forks source link

Incorrect warning when using :withName #2072

Closed antunderwood closed 3 years ago

antunderwood commented 3 years ago

Bug report

Expected behavior and actual behavior

When specifying e.g withName: PROCESS_NAME within a named workflow, Nextflow complains 

WARN: There's no process matching config selector: PROCESS_NAME

However the directive is actioned.

Changing this to withName: WOPRKFLOW_NAME:PROCESS_NAME removes the warning but the directive is ignored.

Steps to reproduce the problem

Add a process to a named workflow and specify the withName directive for the process in nextflow.config

Program output

WARN: There's no process matching config selector: PROCESS_NAME

Environment

Additional context

antunderwood commented 3 years ago

Cloning this repo https://github.com/nf-core/bactmap.git and running it with

NXF_VER=21.04.0-edge nextflow run main.nf -profile conda,test

will give the warnings

WARN: There's no process matching config selector: BWA_INDEX
WARN: There's no process matching config selector: SAMTOOLS_SORT
drpatelh commented 3 years ago

Yup. Sorry, been meaning to report this for a while. I pinged you on Slack @pditommaso but I think you may have missed it amongst your everyday battles 😅

Copying the messages here: If I have defined a name selector in the base.config of the pipeline like here I still get the warning below even though a process with that name exists in the pipeline as you can see here. We have come across it quite a few times with users on nf-core trying to overwrite the default configs shipped with pipelines and the warning is quite misleading because if specified correctly the withName statement has actually been recognised by NF as expected.

image (5)

pditommaso commented 3 years ago

I think you may have missed it amongst your everyday battles 😅

Think the same 😆

Need to check, I'll give a look at my earliest convenience

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

pditommaso commented 3 years ago

Bump

drpatelh commented 3 years ago

Bump bump! We are shifting to using withName as process selectors in the next iteration of the nf-core DSL2 implementation but getting alot of these warnings now as a result which will confuse the hell out of anyone running the pipelines.

image

pditommaso commented 3 years ago

never a good news

abhi18av commented 3 years ago

I have also looked into this earlier and this seems to be happening (at least in my case) because of the derived names in DSL2 i.e. WORKFLOW:PROCESS_NAME - the current mechanism doesn't seem to accomodate the presence of :.

mahesh-panchal commented 3 years ago

Might this be a solution? Change item.name to item.processName or item.getName() here https://github.com/nextflow-io/nextflow/blob/65e2a4d056d8514ae427ac6992c6f06cc13d1271/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy#L211-L218 The logs from the line https://github.com/nextflow-io/nextflow/blob/65e2a4d056d8514ae427ac6992c6f06cc13d1271/modules/nextflow/src/main/groovy/nextflow/Session.groovy#L795 indicate that only the simple process name is being retrieved.

Sep-28 20:01:39.093 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: BAR, FOO
Sep-28 20:01:39.114 [main] WARN  nextflow.Session - There's no process matching config selector: ALEVEL:.*:(FOO|BAR)
Sep-28 20:01:39.179 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:ALEVEL:.*:(FOO|BAR)` matches process ALEVEL:DO_STUFF:FOO

So always https://github.com/nextflow-io/nextflow/blob/65e2a4d056d8514ae427ac6992c6f06cc13d1271/modules/nextflow/src/main/groovy/nextflow/Session.groovy#L872 always returns the warning.

pditommaso commented 3 years ago

Can anybody provide a minimal repeatable test?

abhi18av commented 3 years ago

Here's what I have been testing with 

nextflow.enable.dsl = 2

process download {
    script:
    """
    echo "getReferenceGTF:download"
    """
}

process index {
    script:
    """
    echo "getReferenceGTF:index"
    """
}

workflow getReferenceGTF {
    download()
    index()
}

process {
    // withName:'getReferenceGTF:index' {
    //     cpus=1
    // }

    // withName:"getReferenceGTF:down*" {
    //     cpus=1
    // }

    // withName:"getReferenceGTF:*" {
    //     cpus=1
    // }

    withName:"getReferenceGTF.*" {
        cpus=1
    }

    // withName:".*index.*" {
    //     cpus=1
    // }

    // withName:".*download.*" {
    //     cpus=1
    // }

}

Then in the logs contradicting lines are observed and the config isn't applied.


Oct-13 12:30:01.375 [main] WARN  nextflow.Session - There's no process matching config selector: getReferenceGTF.*
Oct-13 12:30:01.403 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:getReferenceGTF.*` matches process getReferenceGTF:download
Oct-13 12:30:01.702 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:getReferenceGTF.*` matches process getReferenceGTF:index
mahesh-panchal commented 3 years ago

main.nf:

#! /usr/bin/env nextflow

nextflow.enable.dsl = 2

workflow {

    main:
    DO_STUFF('hello')

}

workflow DO_STUFF {

    take:
    messages

    main:
    FOO(messages)
    BAR(messages)

}

process FOO {

    input:
    val str

    script:
    """
    echo $str
    """
}

process BAR {

    input:
    val str

    script:
    """
    printf "%s\n" $str
    """
}

nextflow.config:

process {

    withName: 'DO_STUFF:(FOO|BAR)' {
        echo = true
    }

}

Command: nextflow run main.nf Output: 

N E X T F L O W  ~  version 21.04.0
Launching `main.nf` [modest_morse] - revision: 6ac817e89f
WARN: There's no process matching config selector: DO_STUFF:(FOO|BAR)
executor >  local (2)
[e4/e536da] process > DO_STUFF:FOO [100%] 1 of 1 ✔
[f1/2912c9] process > DO_STUFF:BAR [100%] 1 of 1 ✔
hello

hello

The selector clearly matches, but reports it doesn't. Relevant log:

Oct-13 12:53:04.298 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: BAR, FOO
Oct-13 12:53:04.339 [main] WARN  nextflow.Session - There's no process matching config selector: DO_STUFF:(FOO|BAR)
Oct-13 12:53:04.370 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:DO_STUFF:(FOO|BAR)` matches process DO_STUFF:FOO

Process simple name is printed to log, warns no matching selector, but ProcessConfig finds a match.

pditommaso commented 3 years ago

I've uploaded a patch solving this 4d7d3e972. Could you please make a try to verify it solves the problems on your end? 

NXF_VER=21.10.0-SNAPSHOT nextflow run .. etc
mahesh-panchal commented 3 years ago

I cannot retrieve the snapshot:

NXF_VER=21.10.0-SNAPSHOT nextflow run main.nf -profile test,docker
Downloading nextflow dependencies. It may require a few seconds, please wait .. curl: (22) The requested URL returned error: 403

and 

wget https://www.nextflow.io/releases/v21.10.0-SNAPSHOT/nextflow-21.10.0-SNAPSHOT-one.jar
--2021-10-15 10:53:21--  https://www.nextflow.io/releases/v21.10.0-SNAPSHOT/nextflow-21.10.0-SNAPSHOT-one.jar
Resolving www.nextflow.io (www.nextflow.io)... 54.230.96.75, 54.230.96.87, 54.230.96.65, ...
Connecting to www.nextflow.io (www.nextflow.io)|54.230.96.75|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-10-15 10:53:21 ERROR 403: Forbidden.
pditommaso commented 3 years ago

Oops.. please, try it again

mahesh-panchal commented 3 years ago

Same message

pditommaso commented 3 years ago

Umm, this works for me 

» NXF_VER=21.10.0-SNAPSHOT nextflow info
  Version: 21.10.0-SNAPSHOT build 5634
  Created: 15-10-2021 08:49 UTC (10:49 CEST)
  System: Mac OS X 10.15.7
  Runtime: Groovy 3.0.9 on Java HotSpot(TM) 64-Bit Server VM 11.0.8+10-LTS
  Encoding: UTF-8 (UTF-8)
mahesh-panchal commented 3 years ago 
NXF_VER=21.10.0-SNAPSHOT nextflow info
Downloading nextflow dependencies. It may require a few seconds, please wait .. curl: (22) The requested URL returned error: 403
ERROR: Cannot download nextflow required file -- make sure you can connect to the internet

Alternatively you can try to download this file:
    https://www.nextflow.io/releases/v21.10.0-SNAPSHOT/nextflow-21.10.0-SNAPSHOT-one.jar

and save it as:
pditommaso commented 3 years ago

Think it was a temporary cloud cache problem, please try it again 😬

mahesh-panchal commented 3 years ago

Boom! Thank you:

(nextflow-env) IMB-C02PG08LG3QP:test_selector_labels mahpa906$ nextflow run main.nf 
N E X T F L O W  ~  version 21.04.0
Launching `main.nf` [angry_lamarr] - revision: 6ac817e89f
WARN: There's no process matching config selector: DO_STUFF:(FOO|BAR)
executor >  local (2)
[e0/096933] process > DO_STUFF:FOO [100%] 1 of 1 ✔
[05/6a4b69] process > DO_STUFF:BAR [100%] 1 of 1 ✔
hello

hello

(nextflow-env) IMB-C02PG08LG3QP:test_selector_labels mahpa906$ NXF_VER=21.10.0-SNAPSHOT nextflow run main.nf 
N E X T F L O W  ~  version 21.10.0-SNAPSHOT
Launching `main.nf` [cheeky_fermat] - revision: 6ac817e89f
executor >  local (2)
[c6/298921] process > DO_STUFF:FOO [100%] 1 of 1 ✔
[48/995b77] process > DO_STUFF:BAR [100%] 1 of 1 ✔
hello

hello

(using test script above)

mahesh-panchal commented 3 years ago

( It works)

pditommaso commented 3 years ago

:v: