legsem / legstar-pdi

Automatically exported from code.google.com/p/legstar-pdi
0 stars 2 forks source link

Add parallel processing capability #4

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
This should allow multiple copies of the step to process part of a file 
concurrently.

Very large files can hence be processed faster on multi-core or clustered 
servers.

The difficulty is to find a way to segment the input file evenly between step 
copies in cases where records are variable length (frequent on mainframes).

Original issue reported on code.google.com by fady.mou...@gmail.com on 21 Nov 2010 at 8:07