search

uwsampa / grappa

Grappa: scaling irregular applications on commodity clusters
grappa.io
BSD 3-Clause "New" or "Revised" License
159 stars 51 forks source link

Buffered Parallel file IO by line #84

Open bmyerz opened 11 years ago

bmyerz commented 11 years ago

In order to support buffered reads through Grappa file IO of line-based text files.

If we put the FileIO library in charge of parallelization--as in, for example, the Grappa_read_array API--then the user will provide a callback on_line( char * line).

An ordered line-based interface has to provide also a line index (on_line( int64_t i, char * line)), but requires an extra pass through the file to do a prefix sum. A reasonable simplification of this might be to accept only normalized files where all lines are equal length.

bmyerz commented 10 years ago

Favor fixed-length record format for parallel accessed files