search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

--hdfs-download-source and spark-download-source {v} formatting requires --hdfs-version/--spark-version to be specified earlier #265

Closed jkgeyti closed 4 years ago

jkgeyti commented 6 years ago

It's very convenient that we can use {v} to template the sparh/hdfs version in --hdfs-download-source http://path.to/hadoop_{v}.tgz or --spark-download-source http://path.to/spark_{v}.tgz. However, the only case where this works, is if the --hdfs-version or --spark-version has been set earlier on the command line (i.e. order of arguments matter).

More importantly, the values in the config files are not using for {v} templating.

I know the {v} templating is is an undocumented feature, but it'd be nice if it played well with all config sources, as it's an incredibly useful way to provide our own mirrors to our data scientist, who only have to specify --hdfs-version or --spark-version that way.

Before taking a stab at a PR, i'd like just like to briefly discuss if all of this is even the intended behavior, or if something else is in the works wrt to mirrors etc.

nchammas commented 6 years ago

Hi @jkgeyti. Please provide the information requested in the issue template.

Please also provide some example invocations of Flintrock with example config files to better illustrate the issue you are experiencing. I am not clear on what you are seeing.

nchammas commented 5 years ago

Any update here on the information I requested in my previous comment, @jkgeyti?

nchammas commented 4 years ago

Closing this issue because I don't have enough information to act on it. Feel free to reopen it @jkgeyti with the information I requested.