saga-project / BigJob

SAGA-based Pilot-Job Implementation for Compute and Data
http://saga-project.github.com/BigJob/
Other
19 stars 8 forks source link

Allow one CU to write paritial set of output files to Multiple different DUS #172

Closed pradeepmantha closed 10 years ago

pradeepmantha commented 10 years ago

Use case - In case of MapReduce - where each Map tasks generates set of map output files - one for each reduce.

Currently all these files are stored together in a DU, and later all the files related to a reduce from Map tasks are segregated by the MapReduce framework to pass the DU's as inputs to reduce task.

This could be optimized, if we can allow a Map task to write output files to multiple reduce DU's directly.

I envison, this could be a useful feature for other use cases too.

pradeepmantha commented 10 years ago

This is already working, verified with develop branch.