nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.71k stars 620 forks source link

Implicit variable launchDir breaks caching #5253

Open Sodbo opened 1 month ago

Sodbo commented 1 month ago

Bug report

Ciao from Human Technopole, Milano!

Expected behavior and actual behavior

Using the $launchDir variable in the script: section breaks caching (the process gets executed even though nothing has changed).

Steps to reproduce the problem

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

process test_cache_launchDir {
    output:
        path("result.txt"), emit: result

    script:
    """
    file ${launchDir}/foo.txt > result.txt
    """
}

process test_cache_projectDir {
    output:
        path("result.txt"), emit: result

    script:
    """
    file ${projectDir}/foo.txt > result.txt
    """
}

workflow {
    test_cache_launchDir()
    test_cache_projectDir()
}

Program output

[bugtest]sodbo.sharapov@hnode02$ nextflow run main.nf

Nextflow 24.04.4 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [pedantic_bose] DSL2 - revision: a61cfacf8f

executor >  local (2)
[8f/62d857] process > test_cache_launchDir  [100%] 1 of 1 ✔
[d3/32cc2a] process > test_cache_projectDir [100%] 1 of 1 ✔

[bugtest]sodbo.sharapov@hnode02$ nextflow run main.nf -resume

Nextflow 24.04.4 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [sleepy_kalman] DSL2 - revision: a61cfacf8f

executor >  local (1)
[39/6cbe9d] process > test_cache_launchDir  [100%] 1 of 1 ✔
[d3/32cc2a] process > test_cache_projectDir [100%] 1 of 1, cached: 1 ✔

Environment

Additional context

A similar problem with projectDir was fixed three years ago in Issue #2186. The fix was implemented in commit 7a4d1f1.

I also checked the task hashes, and they differ between same runs, as it was reported here.

bentsherman commented 1 week ago

Even though a patch was made for projectDir, I would argue that neither of these variables should be used directly in the process definition in the first place. You should provide those files as process inputs, then they will be cached correctly

pditommaso commented 1 week ago

Good, point it would be nice the lang server linter reports a warning for it