Event stream table size limit?

[suzdalnitski]

Given enough rows in a relational database table, the performance is likely going to start degrading. How does eventstore handle this internally? Does it make use of table partitioning?

What is the scale that eventstore can handle?

For example, if an application appends 100 events per second, this is going to result in 260,000,000 rows in a month (and about 3 billion in a year). Is this something that eventstore is going to be able to handle?

[suzdalnitski]

I did an experiment to answer my own question - I've been appending events for a few days, and measuring the time it took to append an event. I got to about 600m events before I started running out of disk space, and here are the results (for the append operation):

And the mean:

It seems that the append operation becomes fairly linear after about 100m events:

• At 1m total events it takes 72 usecs to append one event
• At 10m it takes 90 usecs
• At 100m it takes 136 usecs
• At 200m it takes 220 usecs
• At 300m it takes 329 usecs
• At 400m it takes 398 usecs
• At 500m it takes 493 usecs
• At 600m it takes 550 usecs

Extrapolating forward, it might take about 1ms to append one event at 1bln total events, and 10ms at 10bln events.

In my own experiments with Commanded, dispatching an event takes about 15ms (strong consistency, including rebuilding the read models), so 1ms @ 1bln is negligible, however the difference may become noticeable once we get to about 10bln events. Of course, Postgres can handle much more, especially with partitioning/AuroraDB.

@slashdotdash I guess the tables can't be easily partitioned, so sharding probably is the preferred approach with a large enough number of events?

[dvic]

Nice benchmark! Quick question: did you batch insert or insert one event at a time?

[suzdalnitski]

Thanks! I've been using EventStore.append_to_stream/3, appending 1000 events to a stream at a time.