search

SocketCluster / sc-crud-rethink

Realtime CRUD data management layer/plugin for SocketCluster using RethinkDB as the database
33 stars 3 forks source link

Performance #2

Open zalmoxisus opened 8 years ago

zalmoxisus commented 8 years ago

In current implementation to get N items per page we make N+1 queries. So, with first read request we get the items ids, and then request data for every item. Why not to get all the fields with one request?

jondubois commented 7 years ago

@zalmoxisus Sorry for the massive delay in response ;p (for some reason my GitHub notifications were turned off).

That's a good question. Ultimately, it's to reduce complexity on the front-end. To be more specific:

If we loaded an entire item and subscribed to changes on the whole item (instead of changes to individual fields), especially for a large item that has a lot of fields, there would be a lot of wasted change notifications for fields that we don't use (sometimes very large notifications containing very long strings) - We only want to get change notifications for fields that we actually use on our front-end.

With large apps, you might have large documents with 100 fields but only show maybe 10 of those fields on the front-end. We could do some optimisations to only get notifications for fields that are actually used but that solution would be complex and error-prone.

Having very fine-grained models (based on fields instead of documents) basically allows us to be very specific about what data we need on the front-end. It's like working with lots of small Lego blocks that are all the same size - It's easier to work with (and more flexible) than a few big blocks with different shapes.

For new versions of sc-crud-rethink, a while ago, I added caching on the server-side - This keeps the whole document in memory for a while (and that memory cache still gets updated real-time) - So it only does a single DB lookup when you send multiple WebSocket requests for multiple fields in the same document. WebSocket requests themselves are cheap so the DB lookup was the main cost there.

I could probably have made it so that the model would read a subset of all the fields at once but only subscribe to changes on each field individually maybe - That would be a little bit more efficient maybe - It's a little bit harder to model this in Polymer.