robsyme
commented
2 months ago Here is a self-contained example:
workflow {
input = [
[ id: 'test1', patient: 'patient_1' ],
[ id: 'test2', patient: 'patient_3' ]
]
Channel.fromList(input)
| map { patient -> [patient.id, patient] }
| set { patients }
patients
| map { id, patient -> id }
| createFile
| join(patients)
| map { id, initialFile, patient -> [patient, initialFile] }
| addPatientInfo
| view
}
process createFile {
input:
val id
output:
tuple val(id), path("${id}.txt")
script:
"""
touch ${id}.txt
echo "ID: ${id}" >> ${id}.txt
"""
}
process addPatientInfo {
publishDir 'publish', mode: 'copy'
input:
tuple val(meta), path(initial_file)
output:
path "${meta.id}_with_patient.txt"
script:
"""
cp ${initial_file} ${meta.id}_with_patient.txt
echo "Patient: ${meta.patient}" >> ${meta.id}_with_patient.txt
"""
}If we were able to specify in the createFile which pieces of metadata were really relevant, this could be considerably shorter without sacrificing the reusability if the patient value changes for one or more samples:
workflow {
input = [
[ id: 'test1', patient: 'patient_1' ],
[ id: 'test2', patient: 'patient_3' ]
]
Channel.fromList(input)
| createFile
| addPatientInfo
| view
}
process createFile {
cache { meta.id }
input:
val(meta)
output:
tuple val(meta), path("${meta.id}.txt")
script:
"""
touch ${meta.id}.txt
echo "ID: ${meta.id}" >> ${meta.id}.txt
"""
}
process addPatientInfo {
publishDir 'publish', mode: 'copy'
input:
tuple val(meta), path(initial_file)
output:
path "${meta.id}_with_patient.txt"
script:
"""
cp ${initial_file} ${meta.id}_with_patient.txt
echo "Patient: ${meta.patient}" >> ${meta.id}_with_patient.txt
"""
}
bentsherman
pditommaso
wangyang1749
SPPearce
New feature
One of Nextflow's most useful features is the ability to send metadata though the process DAG in parallel to the file data. Not all of the metadata will be necessary to all processes, but any change to the metadata will result in a new task hash, and potentially unnecessary recomputation of tasks.
I propose that users would appreciate the ability to signal to Nextflow which parts of the metadata are relevant to the process.
Usage scenario
Let's say we have a samplesheet
samples.csvThat we ingest in a simple workflow
If we expand the samplesheet to include
employermetadata:Any resumed runs will be unable to use the previously-computed tasks, as the GetEducation
metainput has changed, even if we know that the new input will make no difference to the task calculation.Suggest implementation
I propose a new option to the
cachedirective, whereby the user could supply a closure that returns some new value to be used as input into the task hash:This would would expand the opportunity to rely on the Nextflow task cache when a workflow is being interactively developed, and the extend of the required metadata is not known at the start of development.
The syntax might also include tuples and files (or bags of files):