Spark 2 compatibility - Githubissues

databricks / reference-apps

Spark reference applications

Other

656 stars 341 forks source link

Open tashoyan opened 8 years ago

tashoyan commented 8 years ago

As Spark 2 is not a breaking news anymore, it's time to make reference apps working with it. I can see at least the following things to do:

Use SparkSession instead of SQLContext
Use Dataset API
Update JavaDStream.foreachRDD() invocations to new contract (returns nothing)
Update JavaPairDStream.updateStateByKey() to new contract (uses Spark implementation of Optional)

tashoyan commented 8 years ago

I can handle Java part. The question: do we still need to maintain Java 6 compatible code? Can we exclude support for this outdated Java version?