search

jetoile / hadoop-unit

Hadoop-Unit is a project which allow testing projects which need hadoop ecosysteme like kafka, solr, hdfs, hive, hbase, ...
https://blog.jetoile.fr/hadoop-unit/
Apache License 2.0
51 stars 15 forks source link

Update Spark version from 1.6 to 2.2 #83

Closed martmull closed 6 years ago

martmull commented 6 years ago

Hi, The renault datalake updated its Spark version from 1.6 to 2.2 Do you think you can update the Spark version on hadoop unit standalone version too? Thanks,

jetoile commented 6 years ago

Hello,

Hadoop Unit does not provide spark. If you want to use spark, you have the local mode. Hadoop Unit just provides a pom’s helper (hadoop-unit-client-spark) with which you can use to avoid manually adding exclude section in the pom.xml. By the way, Hadoop Unit is already using Spark 2.2.

martmull commented 6 years ago

Ok, thank you jetoile.