Storage support for Hazelcast

[neilstevenson]

Would storage support for Hazelcast IMDG (imdg.hazelcast.org) be useful ? If so, I could submit a PR.

Being in-memory, this would be similar to zipkin.storage.type=mem, except the memory storage would be on one or more external Hazelcast processes, not part of the Zipkin server itself.

This would give scalability and resilience, without impacting on performance. And optionally, it would be easy also to provide expiry, as a configuration option.

[codefromthecrypt]

word of caution is that storage side tends to be the most code and maintenance. Usually we wait for multiple folks to show interest, and of course you could pilot on your own in the mean time.

[shakuzen]

I think Hazelcast as a storage option is quite interesting, but I will also echo what @adriancole stated. I think this would do better, at least initially, outside of the main Zipkin repository. Once it is off the ground there with some users and support, it might be a good time to consider merging it in as an official storage option. I'd be willing to help review an implementation as well, if useful.

[codefromthecrypt]

PS please do add me if you make a repo, I can help review etc

[codefromthecrypt]

just tweeted in efforts to call attention to the offer for help on this. looking forward to what comes of it.

[neilstevenson]

I've already started, I shouldn't need help. If you get volunteers, get them to pick something else off the wishlist to progress that. PR to follow soon hopefully.

[neilstevenson]

I have an implementation in https://github.com/neilstevenson/zipkin/tree/hazelcast. All tests pass, but I don't think it's quite right, so I'll have to add some more to the unit tests too :-)

[codefromthecrypt]

Thanks for giving a swing and impressive getting all the tests to pass.

Did a quick look and have the following thoughts

• is it possible to not use XML aka check in XML files? I suspect java config will be less indirect for the storage component, and also reduce confusion when binding to property based configuration in spring. Can we use java config to supply defaults instead of XML?
• I'd advise against doing merge logic on consumption (implying read-backs). Typically we merge at query time, and in doing so functionality such as "raw trace" (what instrumentation sent) is retained. This also allows certain things like timestamps to remain authoritative vs guessing.
• On timestamps, I'd suggest adding things with an ingest timestamp if one is missing from the span (but leaving the span in-tact). You'll notice other storage stuff walking around accidentally setting an improper timestamp on a span.

[neilstevenson]

Pragmatically, how would you like to proceed ? A PR to a branch on the main repo, then start addressing, or address in my repo first ? Either way, I guess we'd need a list of mandatory and optional suggestions

[codefromthecrypt]

Regardless, it is probably best to start with a separate repo as there's still no user interest noted here despite tweeting, mentioning this in release notes, etc. We have to conserve energy in this repo for diversely needed change.

I'd suggest making that repo integrated as a library with dependencies vs forking this repo. If you want a special server to make testing easier in the short term, you could consider making something similar to stackdriver-zipkin. Otherwise, you can do a pure library integration like done in zipkin-aws or zipkin-azure.

This repo should be in your account or org; in github it is easy to move repos when the time comes to consider such. Once there, you can feel free to add me or tommy who can check out things as we can.