HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

Moar testing: Run against Spark 2 and 1.5 with docker, run test-all in CircleCI #21

Closed viesti closed 7 years ago

viesti commented 7 years ago

Seems that Spark 1.5 works with kryo 3.0.3 while Spark 2 needs kryo 4.0.0. Also use :target-path "target/%s/" just to make sure that compiled classes are aware of active profile.

I'm using https://hub.docker.com/r/gettyimages/spark/ for as docker image and while 2.1.0 seems to work neat, when using 1.5.2, one needs to have master name mapped to 127.0.0.1 in /etc/hosts.