search

thunder-project / thunder

scalable analysis of images and time series
http://thunder-project.org
Apache License 2.0
814 stars 184 forks source link

Update for Travis CI to test against spark 2.0 #360

Closed boazmohar closed 8 years ago

boazmohar commented 8 years ago

Address #359

jwittenbach commented 8 years ago

Well look at that -- awesome! 👍

freeman-lab commented 8 years ago

Well that was straightforward 🎉 thanks @boazmohar ! Can you add a note to the README as part of this PR saying that it works with Spark 2.0.0?

boazmohar commented 8 years ago

Yep, was very easy. I guess the parts that changed a lot in 2.0 are DataFrame and Dataset not RDD's.

jwittenbach commented 8 years ago

I've been thinking that it would be test whether there might be any speed-up in switching things to DataFrames -- though I haven't had any time to play with it.

freeman-lab commented 8 years ago

Yeah agreed @jwittenbach, we should definitely look into that, I think the Dataset API may be what we want to look at. But in the meantime let's merge this in, good to make clear it's working with the latest version.