search

keenlabs / KeenClient-Java

Official Java client for the Keen IO API. Build analytics features directly into your Java apps.
https://keen.io/docs
MIT License
74 stars 43 forks source link

Some questions about queueEvent and sendQueuedEvents #21

Closed facundofarias closed 10 years ago

facundofarias commented 10 years ago

Let's suppose we have KeenIO on our infrastructure up and running on my backend, and then I can access the Workbench, or extract the data for analysis, which it's quite good. The thing is, we've noticed some outages on the Data Collection API during August (http://status.keen.io/#month), which are really really small (Availability of 99.960%), but we've lost some events on our way. So, we are thinking on different approaches to solve this issue: Does the SDK provide some kind of mechanism to retry? Or, should we write on our side? I've seen you have the method queueEvent, could we use the callback to detect if an error occurs, and then put it on the queue?

Geeber commented 10 years ago

Hi! If you're using the addEvent method (or its async counterpart) then there isn't any built-in retry logic. You can use the callback mechanism to detect failures and add your own retry logic if you want to continue using addEvent. An alternative would be to switch to using queueEvent/sendQueuedEvents, which sends a batch of events in a single HTTP request. sendQueuedEvents parses the response from the server and only removes events from the queue if they are successfully acknowledged. So if you are regularly calling sendQueuedEvents (e.g. on a timer) then it can act as a retry mechanism.

Also, just a note that we are currently discussing ways to provide more control over retry logic. We'd be happy to hear any feedback or feature requests! And if you'd like to discuss your situation in more detail, just let me know and we can set something up.

facundofarias commented 10 years ago

Good. So perhaps it's a good idea to use the sendQueuedEvents instead of the addEventAsync for our needs (although it is not real-time anymore, but that does not matter). So, when you say To store events in a queue and periodically post all queued events in a single batch, use the queueEvent and sendQueuedEvents (or sendQueuedEventsAsync) methods on the documentation, which will be that periodicity!? I mean, every how much time are you sending those events on that queue? Is it configurable?

Thanks! :+1:

Geeber commented 10 years ago

queueEvent will add a single event to the queue, and sendQueuedEvents will attempt to send all currently queued events. So the period is just however often you decide to call sendQueuedEvents. For example, you could run a background thread that just sleeps for 60 seconds, calls sendQueuedEvents, then loops. Or you could attach it to some regular event in your application's life-cycle, if that's easier.

As background: the primary reason we provided the queue/batch capability in the first place was to support Android, where it's bad for battery life to be making constant network connections. Typically Android apps would call sendQueuedEvents when the application is started or stopped, for example. If you're using the SDK from a long-running Java server process, then a simple timer thread may be the best option.

One other note to be aware of is choosing between storing queued events in RAM vs. on your filesystem. RAM will be faster, but obviously isn't persistent, which you may or may not care about. And you will want to be careful about how many events you let accumulate in the queue; for now we default to a maximum of 10,000, after which we start aging out old events.

Hope that helps!

Geeber commented 10 years ago

As I think your question has been answered, I'm closing this issue. Please feel free to re-open if you have follow-up questions. Thanks!

facundofarias commented 10 years ago

Yes @Geeber thanks a lot! :+1: