search

openzipkin-attic / zipkin-azure

Reporters and collectors for use in Azure
Apache License 2.0
11 stars 9 forks source link

Deal / Document with Event Hubs limits #28

Open clehene opened 7 years ago

clehene commented 7 years ago

There's a hard limit of 20K messages per namespace. This may limit a users ability to use this with larger infrastructure.

Here are some details about quotas / limits https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-what-is-event-hubs

aliostad commented 7 years ago

Some discussions in issue #30 related to this.

In short: If needs be, dedicated EventHub available that can do up to 2M event/second. Also DocumentDB is not really a replacement, for cheaper option one could go for Kafka as IaaS on Azure.

aliostad commented 7 years ago

@clehene are you happy with the explanation?

gambrose commented 7 years ago

@clehene can you elaborate where you get the 20K limit from?

I have quite recently started using event hubs. I needed to push a lot of small events and as there is a cost per event the costs would have been large. So I just batched my application events inside an event hub event. I also compressed the event payloads.

I don't know the size of the messages you are pushing but batching was at least an order magnitude cheaper for us.