Closed percyfal closed 11 years ago
Use pipes whereever possible. See luigi.format.InputPipeProcessWrapper etc and luigi.file. Incidentally, how would using pipes over several nodes work? I guess tasks connected by pipes should be gathered and run by a single worker
Use pipes whereever possible. See luigi.format.InputPipeProcessWrapper etc and luigi.file. Incidentally, how would using pipes over several nodes work? I guess tasks connected by pipes should be gathered and run by a single worker