mskcc / tempo

CCS research pipeline to process WES and WGS TN pairs
12 stars 5 forks source link

use Channel.interval() factory instead of watchPath #1009

Open anoronh4 opened 6 months ago

anoronh4 commented 6 months ago

See documentation here: https://www.nextflow.io/docs/edge/channel.html#interval

Currently we're using a workaround to use the watchPath function in dsl2. this workaround still has limitations, as it requires us to touch -ca the file just to read it. this also has limitations, for example if we do not own or have permission to touch -ca.

This new factory should be able to replace the workaround. We should basically be able to set an interval and with each emission read an input file. This feature only seems to be available in the edge version right now.

for further context about why we needed a workaround in the first place, watchPath doesn't work by itself because it uses inotify, which requires the target of watchPath to be modified on the same machine as where nextflow is running. so if something is modified in one node of a cluster, and watchPath is running in another node, nothing will happen.

anoronh4 commented 5 months ago

looks like watchPath doesn't work with touch -ca in the new version of nextflow anyways. the following doesn't trigger watchPath:

new Timer().schedule({
def cmd = "touch -ca hi" 
cmd.execute()
} as TimerTask, 1000, 5000 )
Channel.watchPath(file("hi"),'create,modify').view()

it works as late as 22.04.5 but stops working as early as 22.10.3. it also doesn't work with 24.03.0-edge, which is pretty recent.