Parquet integration - Githubissues

gorillalabs / sparkling

A Clojure library for Apache Spark: fast, fully-features, and developer friendly

https://gorillalabs.github.io/sparkling/

Eclipse Public License 1.0

448 stars 68 forks source link

Parquet integration #58

Closed LukeDefeo closed 7 years ago

LukeDefeo commented 7 years ago

Hey guys we have extended the data frame implementation with support for:

Parquet reading writing
clojure api for creating struct types / struct fields (schema definition for data frames)
a few helpers for data frames

BrunoBonacci commented 7 years ago

Nice work. I've tested this PR and it works nicely. I've used it to encode 15 billion records into parquet format via the data-frames. Thanks @LukeDefeo.

When are you guys planning to review/merge this PR?

BrunoBonacci commented 7 years ago

👍

obohrer commented 7 years ago

👍

johntelforduk commented 7 years ago

👍

chrisbetz commented 7 years ago

Thanks for the input! I've been really busy the last month, sorry for that.

I really like your contribution! I'm merging today and will make a new version soon. I'm at Euroclojure today, so will take some days.

chrisbetz commented 7 years ago

Done. Released from Euroclojure: Hope you're happy to go with sparkling 2.1.0 :)