Latency critical approach (question)

[ghost]

I'm building a algotrading tool, for that i'm using ES + CQRS (ability to track down any issues in my algorithms is required). And i've recently tried EventStore (also in memory mode), its a brilliant one of the kind tool and way beyond simple datastore, but for my specific case i need very short latency with datastore as it is going to affect my algorithms efficiency...

The approach i saw in eventstore projectsions is very similar the one of reactive programming and observables... i got sources of data (streams) and i can do different transformations on them and have new source of data in the end...

So my question is, if i have a latency critical task i want to do with event stream (and i dont really need that data to be stored) do you guys think it is viable solution to use Rx for latency critical part (so that i dont wait database IO) and let the other non critical (ui related) streams be handled with event store?

I'm asking because i dont have much experience with ES, CQRS, EventStore and i might be missing some approach i dont really know of... what do you do when you need a latency critical stream?

[ghost]

suppose you have few streams from eventstore that you really need to have shortest latency what would you do?

P.S. i think it will be very fast if i use LMAX Disruptor (with buzyspin) to paralelize the two tasks [one calling to eventstore] and the other one [using Rx to instantly produce the result].

[jen20]

Likely the best approach is not put them in Event Store, at least not in a synchronous manner. You don't really want to be waiting for a quorum of machines to be writing to disk before your write is acknowledged if you're low-latency sensitive.

[ghost]

Thanks, it seems i'm on right direction, disrupter will paralelize call to event store so eventually write to eventstore will not block my native projection from processing...

[jen20]

What language are you working in?

[ghost]

I'm on java (8), I was thinking what if I could have an embeded js engine like Nashorn in case of jdk8 and have eventstore running in it, pass the data using direct java<-->js api and subscribe to desired stream... all in memory. Someone else might need to use it this way too... it would be lightning fast...

Also there might be an issue with eventstore keeping all data (if we are in memory we wont have that much of a space) but as projections only need to run trough entire stream only first time, and after that they just project new events using previus state (if i understand this part correctly) then it is teoretically possible to have dedicated eventstore keep all events and this in memory one only the last ones...

I think this could be a feature and some system might benefit from it when latency and fast performance are mandatory... (just sharing toughts)

[ghost]

This can be easily done if there is no tight coupling in source of the actuall engine that processes the projections and the rest of the source handling apis... if those are modular i believe i could easily do this... even have this as a configurable option.

[gregoryyoung]

For low latency you definitely do not want to do this kind of things. Just the v8 garbage collector will start causing you issues (every event also needs a json.parse). There is also quite a bit of code around V8 that would need to be handled (many things actually call back into our internal C++ code)

On Mon, Jun 1, 2015 at 4:52 PM, Vachagan Balayan notifications@github.com wrote:

I'm on java (8), I was thinking what if I could have an embeded js engine like Nashorn in case of jdk8 and have eventstore running in it, pass the data using direct java<-->js api and subscribe to desired stream... all in memory. Someone else might need to use it this way too... it would be lightning fast...

Also there might be an issue with eventstore keeping all data (if we are in memory we wont have that much of a space) but as projections only need to run trough entire stream only first time, and after that they just project new events using previus state (if i understand this part correctly) then it is teoretically possible to have dedicated eventstore keep all events and this in memory one only the last ones...

I think this could be a feature and some system might benefit from it when latency and fast performance are mandatory... (just sharing toughts)

— Reply to this email directly or view it on GitHub https://github.com/EventStore/EventStore/issues/495#issuecomment-107490837 .

Studying for the Turing test

[ghost]

Didnt know about C++ under the hood, thats usefull thanks, So its better to use something like Rx i guess

[gregoryyoung]

In general I would say probably. I would consider using something not like RX as well though. Functional reactive stuff tends to produce more garbage than stuff that works in place (leads to a long tail at 99-99.9%). That said. Benchmark Benchmark Benchmark

On Mon, Jun 1, 2015 at 4:57 PM, Vachagan Balayan notifications@github.com wrote:

So its better to use something like Rx?

— Reply to this email directly or view it on GitHub https://github.com/EventStore/EventStore/issues/495#issuecomment-107495026 .

Studying for the Turing test