search

kite-sdk / kite

Kite SDK
http://kitesdk.org/docs/current/
Apache License 2.0
394 stars 263 forks source link

KITE-1101: Added logic to FileSystemPartitionIterator to exclude dire… #434

Closed mkwhitacre closed 8 years ago

mkwhitacre commented 8 years ago

…ctories with data that does not qualify. Extended predicates and StorageKey code to support partial values.

Note this might violate some of the initial designs for StorageKey and the Predicates so I'm open to alternate ways of doing this logic. @rdblue and @rbrush if you have some insight or critique that would be helpful. It is not necessarily the cleanest changes so if you have suggestions on how to clean this up with original design intent that would be helpful.

In some smaller tests I've seen significant improvements in job startup time (1s vs 100s) and a lot less memory pressure on the driver program when kicking off the job.

noslowerdna commented 8 years ago

+1

rbrush commented 8 years ago

+1. Looks good to me.

mkwhitacre commented 8 years ago

Got two +1s. Merging.