search

cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
859 stars 172 forks source link

Avro vs Protobuf #33

Closed simPod closed 5 years ago

simPod commented 5 years ago

Hi, I wonder whether there was a reason for Protobuf msg format and not Avro. In Kafka ecosystem Avro is AFAIK used more (supported by Schema registry etc.). Would Avro make eventually sense for goflow?

Thanks!

lspgn commented 5 years ago

Hi @simPod, No specific reason, since I wrote this software in Go, I started using Protobufs. Sending to Kafka could also be replaced by sending to GRPC. This can also have some advantages if you send to Google Cloud or Clickhouse (also supports Cap'n Proto). I'm not against adding more encoding formats (there's JSON as well) and version 3 allows this more easily. What would be the use-case? Schema-registry? I don't think I will have time in the close future to work on this but I would gladly accept pull-requests!

simPod commented 5 years ago

Yea, schema-registry at least.
I'm using Clickhouse, it supports even Protobuf natively. But Kafka engine is broken there for now so I use my own inserters anyway, not a big deal 🤔 And it already supports tonz of formats so adding Avro might not be so difficult if necessary.

I was just thinking that Kafka is more Avro friendly. Generally Protobuf is cool, I just don't fancy how Google executes its development and Kafka (or Confluent) doesn't seem to care about it in the near future.

Good to know you're not against it. If it becomes a thing for us and we decide to go that direction, we will submit PR again.

Thanks for having goflow OSS ;)