Open schrepfler opened 5 years ago
A few non-func points from me:
Is 2) a problem? I think in this case we care mostly about the publisher API (agent/middleware -> pulsar) which can also be batched/delayed locally (at risk of loss).
No2 is an advantage of Pulsar, not a problem :) Sorry for confusing :)
Also, Kafka's publisher can block on publish, even if async API is used, while afaik there is no such issue in Pulsar
No problem. I think Kafka did mitigate the blocking client a bit in the last releases (just for fairness/completeness of the argument) https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior
Only in the consumer part, but there are still blocking calls in Kafka's async producer :(
if anyone does a spike, do link back for the fans
I see there was a spike of interest, but unsure any outcome. Did anyone do anything?
I think it would be useful to try to add an additional transport to Zipkin which would be Apache Pulsar.
Feature: Description of the feature
Rational Pulsar has several features which I think make it interesting for us:
Integrated SQL query capability. Latest Pulsar has integrated SQL query capabilities based on the Facebook Presto engine. This means it's possible to query data directly from the topic thus reducing the need to have a separate DB tier. Benefit: Potentially simplified deployments. Unknown: Not quite clear how capable is the query engine and would it be suitable for the kind of queries Zipkin makes.
Tiered storage. Pulsar has the capability to offload topics onto long term/cheap storage (ex. S3) without having to estimate ahead of time any size or time based expiration policies. Benefit: Simplify DB management overhead and provide simple way how to enable cheap durable persistence.
Decoupled storage from brokers By decoupling brokers from storage Pulsar (at least on paper) should be easier to scale out. Benefit: Smaller operational overhead, less chances of errors and simplified operations.
Scales to more topics
Multitenancy and Georeplication capabilties. Benefit: Potentially diverse business uses
Pulsar functions. It's possible to implement very low latency triggers based on messages which trigger actions directly in the middleware.
Kafka compatibility mode. Might be possible to use it in Kafka compatibility mode to simplify development (but at that point I'm not sure if it's possible to leverage the other features).
Web Sockets capability It's possible to expose the data stream over web sockets, might be interesting if we want to do fancy stuff in the UI (like real time tracing on a giant ring, I think this view was lost from the old days at Twitter?)
Lower latency thank Kafka (on paper)
Schema registry If the data is persisted in one of the formats that support schema it might be possible to evolve the format by using the format schema tools and the registry. I think using a schema is a requirement in order to be able to use Presto to query the data. JSON, Protobuf and Avro are supported.