Resolve #Issue-5: HDP2.3 & Ambari 2.1 Support

spring-attic / spring-xd-ambari

Apache Ambari integration for Spring XD

Apache License 2.0

9 stars 18 forks source link

Resolve #Issue-5: HDP2.3 & Ambari 2.1 Support #11

Closed tzolov closed 8 years ago

tzolov commented 9 years ago

./gradlew clean dist

produces a new rpm artefact: spring-xd-plugin-hdp23-1.2-2.noarch.rpm

Summary:

Add new build artefact: spring-xd-plugin-hdp23 and build task: hdp23Rpm
Extend the postinstall.sh and preremove.sh to support HDP/2.3 repository structure. NOTE that you could have multiple Stacks installed at the same time. It is not mutually excluding either PHD or HDP or ..
Ambari 2.1 replaces HdfsDirectory by HdfsResource. I've modified to params.py and springxd.py to support the new API and still provide backaward compatibility

jvalkeal commented 9 years ago

Thanks, it sucks that we try to keep same plugin for all ambari versions but then they just rename python methods. I wonder if we should make a tweak to resolve if HdfsDirectory or HdfsResource should be used.

@trisberg do you have comments on this?

jvalkeal commented 9 years ago

ah, ok this PR handles both HdfsDirectory and HdfsResource. Nice!

trisberg commented 9 years ago

Spring XD currently supports HDP 2.2 so maybe we should wait with this PR until we update XD to support HDP 2.3

tzolov commented 9 years ago

@jvalkeal , indeed i've tried to handle the schizmatic HdfsDirectory/HdfsResource change. I've used the the python function to smoothen the transition and keep backward compatibility. I've tested it on small HDP2.3 cluster and it seems to be working. I've tested both manual (Ambari Wizard) and Blueprint API installations. Here is my Blueprint. Note that current implementation uses the Stack version to decide which Hdfs API to use. This is suboptimal as the change is Ambari related (e.g from 2.0 to 2.1 - AMBARI-11034) and should be Stack agnostic.

tzolov commented 9 years ago

@trisberg this is a valid point. It depends though how long it might take for this update to come. Perhaps in the meantime until the update happens we can map internally HDP2.3 to HDP2.2. Have not tested this yet.

trisberg commented 9 years ago

@tzolov I'd rather not go there since we have customers that run a variety of workloads, very difficult to support unless it is officially supported by XD. The plan right now is for XD 1.3 to support HDP 2.3 and PHD 3.3 some time after Spring One which is in September.

tzolov commented 9 years ago

Fair point

jvalkeal commented 9 years ago

ok and kafka config breaks with hdp2.3, so it needs to be fixed first anyway.

jvalkeal commented 8 years ago

closing this pr. @tzolov I picked related changes from this pr to fix for #5, thx mate!

tzolov commented 8 years ago

@jvalkeal YW