search

databio / pypiper

Python toolkit for building restartable pipelines
http://pypiper.databio.org
BSD 2-Clause "Simplified" License
46 stars 9 forks source link

follow functions and process ID determination in pipeline rerunning #151

Closed stolarczyk closed 5 years ago

stolarczyk commented 5 years ago

related to: #142, #144

when a follow function uses run method (like ngstk.check_trim), then it is counted as a process (process ID is incremented) since it uses callprint method BTS. Conversely, follow functions that do not use the run method are not counted as processes (process ID is not incremented).

Is there a way to distinguish between these in command skipping/counting process after restarting the pipeline?

Currently both are treated the same way, which is wrong -- the process IDs in a rerun pipeline do not match the previous ones because the process IDs are incremented for every single follow function, but sometimes they should not be

nsheff commented 5 years ago

can we make it so that any command run as a follow function does not increment the counter? It could be assigned the id of the original command plus "f". this way follow functions in the original run would not increment the counter.