search

onyx-platform / onyx

Distributed, masterless, high performance, fault tolerant data processing
http://www.onyxplatform.org
Eclipse Public License 1.0
2.05k stars 204 forks source link

Upgrade Apache Curator framework to 4.0.x - SSL Support #854

Closed gardnervickers closed 6 years ago

gardnervickers commented 6 years ago

Some users are running hosted Zookeeper, requiring a SSL connection. Curator framework 4 on supports this by using the ZK client libs 3.5.1.

lbradstreet commented 6 years ago

Ugh, the problem with this is that curator 4.0 only supports ZK 3.5+. Maybe we will have to have an exclusions section in our doc?

Thingographist commented 6 years ago

@lbradstreet i tried using CF v4.0 in my project for watching changes in ZK And it work with ZK versions 3.4 and 3.5. However i generally use only ZK 3.4. Dependent project have not problems.

But CF v4.0 allow TTL marks for nodes and it's awesome feature for persistent state of cluster and data flow.

lbradstreet commented 6 years ago

Thanks for the report @Thingographist. I tried bumping curator to v4.0, but it was not compatible with ZK 3.4.11 running via docker (threw some compatibility errors). Did you test it against an external ZK? I would love to upgrade it, but unless it's compatible with 3.4 it will be hard.

Thingographist commented 6 years ago

For development i run cluster on dockerised ZK v3.4.11. For staging - ZK v3.4.5 on root OS (from official debian repository). And it's work with CF 4.0.

Just simple fix from official docs:

[[org.apache.curator/curator-framework "4.0.0"]
 [org.apache.curator/curator-recipes "4.0.0"]
 [org.apache.curator/curator-test "4.0.0"]]
 [org.apache.zookeeper/zookeeper "3.4.11"]
 [org.apache.curator/curator-client "4.0.0" :exclusions [org.apache.zookeeper/zookeeper]
 [org.onyxplatform/onyx "0.12.7-1519160025" :exclusions [
          org.apache.curator/curator-framework
          org.apache.curator/curator-test
          org.apache.zookeeper/zookeeper]]]

And it's work without using isZk34CompatibilityMode method.

This fixes disable features of v3.5, but some methods from CF is works (for example: Tree Cache, Persistent TTL Node). And isZk34CompatibilityMode method maybe globally enable/disable 3.5 features.

lbradstreet commented 6 years ago

Awesome! isZk34CompatibilityMode is the key insight that I was missing :). I'll give this a go tonight.

lbradstreet commented 6 years ago

Upgraded in master and 0.13.0 (soon to be released). Closing