search

ccri / cloud-local

Install script for a local 1 node cloud...no excuses folks
17 stars 5 forks source link

Update Spark to 2.x #43

Closed tkunicki closed 7 years ago

tkunicki commented 8 years ago
cne1x commented 8 years ago

Less one minor comment, this looks good to me. Thanks for the upgrade!

djr7m commented 8 years ago

This changed the default spark version but does it still work if I request 1.6.1? I guess I can test it myself too but thought it was a quick question for @tkunicki

tkunicki commented 8 years ago

@djr7m , The pre-PR spark can be restored with

pkg_spark_ver="1.6.1"
pkg_spark_hadoop_ver="without-hadoop"

That being said, i noticed in my testing that our generalized apache mirror selection can pick mirrors that don't have the some spark versions. Some appear to follow a 2x2 strategy, the 2 latest minor releases of the 2 latest major releases. In testing it appears some mirrors only have the 1.6.[23] releases of spark 1.6. So to answer your Q, 1.6.1 might not download but this isn't an issue introduced by this change. (thoughts @jahhulbert-ccri?)

djr7m commented 8 years ago

Its fine if its a problem with the D/L site, no worries. I just wanted to ask if this P/R had made any other changes to its structure where it would freak out do to "spark < 2.x". I think its fine, I was testing anyway

jahhulbert-ccri commented 8 years ago

yeah 1.6.1 isn't on a lot of mirrors

mdzimmerman commented 7 years ago

I apologize for writing to your repo, @tkunicki-- I ran the command fully expecting it was going to fail. In any case all I did was update the branch with the current master.