Closed ewels closed 9 years ago
I think this is due to the logic involved with the way that all downloads have to run in series, but run groups can be kicked off when a set of files have finished downloading? It's pretty confusing. Anyway, should be at least be able to print downloaded filenames to the run file even if they are ignored by the actual download module.
Done in https://github.com/ewels/clusterflow/commit/04d3e4191de4ab9afcb8e3cbc393bef8f5d9f44c
Not tested yet, but should work.
Currently, the downloads module takes URL and filename as parameters on the command line when called. It'd be better to do this as regular filenames in the run file - less parameters, also other modules will know how many files we're starting with.