bsideup
commented
5 years ago A few non-func points from me:
- Pulsar provides truly async client, while Kafka is coming from the blocking world
- Pulsar consumer API is simpler than Kafka's (where if you don't poll you might end up de-assigned from the partition (sic!))
- Running Kafka behind NAT is problematic, while Pulsar can be used with a proxy
- Pulsar SQL is a driver for Presto, not a brand new tech from scratch by a commercial owner of the tech
- Backing up Pulsar clusters is much easier thanks to the tiered storage
schrepfler
codefromthecrypt
I think it would be useful to try to add an additional transport to Zipkin which would be Apache Pulsar.
Feature: Description of the feature
Rational Pulsar has several features which I think make it interesting for us:
Integrated SQL query capability. Latest Pulsar has integrated SQL query capabilities based on the Facebook Presto engine. This means it's possible to query data directly from the topic thus reducing the need to have a separate DB tier. Benefit: Potentially simplified deployments. Unknown: Not quite clear how capable is the query engine and would it be suitable for the kind of queries Zipkin makes.
Tiered storage. Pulsar has the capability to offload topics onto long term/cheap storage (ex. S3) without having to estimate ahead of time any size or time based expiration policies. Benefit: Simplify DB management overhead and provide simple way how to enable cheap durable persistence.
Decoupled storage from brokers By decoupling brokers from storage Pulsar (at least on paper) should be easier to scale out. Benefit: Smaller operational overhead, less chances of errors and simplified operations.
Scales to more topics
Multitenancy and Georeplication capabilties. Benefit: Potentially diverse business uses
Pulsar functions. It's possible to implement very low latency triggers based on messages which trigger actions directly in the middleware.
Kafka compatibility mode. Might be possible to use it in Kafka compatibility mode to simplify development (but at that point I'm not sure if it's possible to leverage the other features).
Web Sockets capability It's possible to expose the data stream over web sockets, might be interesting if we want to do fancy stuff in the UI (like real time tracing on a giant ring, I think this view was lost from the old days at Twitter?)
Lower latency thank Kafka (on paper)
Schema registry If the data is persisted in one of the formats that support schema it might be possible to evolve the format by using the format schema tools and the registry. I think using a schema is a requirement in order to be able to use Presto to query the data. JSON, Protobuf and Avro are supported.