zrlio / crail-spark-io

Fast I/O plugins for Spark
Apache License 2.0
41 stars 14 forks source link

Broadcast support clean up. Multiple changes. #2

Closed animeshtrivedi closed 7 years ago

animeshtrivedi commented 7 years ago

(1) Broadcast files and opened and closed properly. No pending buffered streams linger till the end. (2) Broadcast values can now be cached locally instead in the local BlockManager. Use 'spark.crail.broadcast.useLocalMap' true|false (3) Broadcast has its own serializer now. Currently it only catches the case of Array[Byte], but it will be expanded gradually.