Test sparkling water distro with provided instructions from Michal

[exalate-issue-sync[bot]]

Kevin Normoyle commented: HDP2.1 doesn't come with Spark as part of the hortonworks distro they have instructions for adding it as part of manual config. I'll do that to add spark to our current hdp2.1

http://hortonworks.com/hadoop-tutorial/using-apache-spark-hdp/

Separately, spark project has instructions for compiling to a hadoop installation and installing but we don't want to do that. We won't do that. That seems more homebrew and unlikely a customer will add spark that way.

they also distribute a VM with HDP2.2 and Spark 1.2 Michal says Spark 1.2 won't be available for release till Nov. So HDP2.2. (full release) probably won't be available till Nov.

So the CDH5.2 we have, has Spark 1.1

the first step in "duplicate M* customer exactly" seems to be to install Spark on the current HDP2.1

There is no Ambari way of adding Spark to HDP today, except in the HDP2.2 VM sandbox. So: assuming customer must be sort of sophisticated to add Spark to current HDP2.1

Not exactly sure what mode of running Spark on CDH5.2 we're talking about. There are different ways, that do memory/heap allocation with YARN differently or ??

[exalate-issue-sync[bot]]

Tom Kraljevic commented: This item has been done by several people now.

[hasithjp]

JIRA Issue Migration Info

Jira Issue: PUBDEV-57 Assignee: Kevin Normoyle Reporter: Tom Kraljevic State: Resolved Fix Version: N/A Attachments: N/A Development PRs: N/A