search

radanalyticsio / silex

something to help you spark
Apache License 2.0
65 stars 13 forks source link

Interface for per-file-parallelism with Hadoop FS #27

Open rnowling opened 9 years ago

rnowling commented 9 years ago

For many of the use cases we've seen, parallelism per file versus per line is sufficient performance wise and makes parsing files easier. (e.g. JSON, CSV, etc. file formats which may require keeping track of state from the start of the file is problematic for chunking up files.)

We'd like to provide an API for per-file-parallelism. We'd like to reuse Hadoop's Filesystem interfaces which provides a unified interface to readers for a variety of sources including S3, FTP, HDFS, and POSIX. If using lazy evaluation is possible, we wouldn't have to keep large files in memory. We may also want a way to list and filter on the list of files. (E.g., more than just glob-type expressions.)

rnowling commented 9 years ago

Follow up idea courtesy of @willb: implement the API as a custom RDD and provide a sc.partitionPerFileRDD("~/input") method.

rnowling commented 9 years ago

The SparkContext provides a wholeTextFile() method which returns pairs of (filename, contents). For large files, this may kill performance so the idea of 1 partition per file may offer better granularity.