Templating code prevents custom Spark builds

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.

Apache License 2.0

637 stars 116 forks source link

Templating code prevents custom Spark builds #190

Closed tomerk closed 7 years ago

tomerk commented 7 years ago

Flintrock version: 0.8.0

I am trying to use flintrock to launch & build from a custom Spark git commit / repository. In order to do this, I must set spark-version to None, as git commit and spark version are marked mutually exclusive.

Unfortunately, in core.py the following line errors when spark_version is None: 'spark_short_version': '.'.join(spark_version.split('.')[:2]),

nchammas commented 7 years ago

Thanks for the bug report. I think this might be a regression introduced in #180.

Do you hit this same bug on Flintrock 0.7.0?

tomerk commented 7 years ago

I haven't tried Flintrock 0.7.0, so I'm not sure if this bug occurs there.

I have a fork of Flintrock that I'm using, and I explicitly put in 2.1.0 in this file and it seems to be working there. I did also have to add in a manual mvn package after the make-distribution.sh command in services.py.