How are IDs handled?

[tslocke]

This looks like a great project - I'm looking at using it for a collaborative app I'm working on.

My model has deeply nested structures, but in the client side store and in the database these are flattened out using unique IDs. If an object contains nested objects, I store IDs instead. In the store everything becomes top level.

It seems like this approach would work well with SyncedStore, but I'm wondering how allocating new IDs would be handled, to ensure they are unique across devices.

One solution would be to use UUIDs, but I find it a pain to work with these huge long IDs.

My current solution is to generate client-specific temp-IDs, and when objects are persisted to the server, the server allocates a proper unique ID and sends a mapping (temp-ID -> perm-ID) back to the client, so the client-side store can be updated accordingly. It's working but I don't love this approach. It's easy to make mistakes and introduce bugs.

It seems this must be a common issue with the kinds of apps SyncedStore is intended for. Is there a built-in solution?

Thanks very much.

[YousefED]

Thanks @tslocke!

Because SyncedStore applications are designed to be distributed, and not necessarily need a central server, I think UUIDs are the best approach tbh.

However, if you have a central server, what you could do is create a Yjs / SyncedStore provider that communicates with your server. The server can then apply updates to the document (replace the tempids) and sync back those updates to clients. I think this is the cleanest approach, and would also work if your server needs to augment your datamodel or make other changes to it (migrations, etc).

Sending a mapping of tempID -> permID would also work, but it's a less "scalable" solution imo.

Does this help you?

[tslocke]

Thanks very much for the advice. Your solution to have the server manipulate the document directly and rely on SyncedStore to get those changes back to clients sounds very nice. Does it mean the server must maintain an in-memory copy of the state of every connected client? Or is there a way to just temporarily hold certain documents in memory, make edits, have those synced back to the client, save to the DB and then discard from memory?

I guess I'm not fully groking the model. There is an easy version, where a bunch of clients all have the complete state in memory, and everything stays magically in sync, e.g. a game. That's easy to understand.

But the case that's more typical to most web apps, where there's some huge state in a database somewhere, and a large number of clients are connected, each of which is only interested in a small slice of that global state, it's hard to understand how that works.

[YousefED]

Or is there a way to just temporarily hold certain documents in memory, make edits, have those synced back to the client, save to the DB and then discard from memory?

Exactly, a server implementation could only hold the documents in memory that are being editted (e.g.: an active user is "connected" via websockets, or it could flush the document to the database if no changes have been made for x seconds).

each of which is only interested in a small slice of that global state

This is the key. You typically divide state into documents / stores. For example, one document / store can represent a game, a todo list, a spreadsheet, a word document, etc. Then, clients can make changes to that document (which can hold multiple data structures / properties), and sync that with each other or via a coordinating server.

What kind of data are you trying to model? Perhaps I can help think along!

[tslocke]

That's a kind offer which I really appreciate. The CRDT world is such a different way of thinking about things, it's not so easy to get started.

A simple way to describe the app is a kind of notes app with wiki-links, but one important feature is the ability to support very large pages, where you can "zoom in" on any sub-section. The page structure is a hierarchy of blocks, and all communication between client and server is at the block level. If the client needs to open a sub-section, the bigger page in which it lives is not transmitted.

So to handle the entire (possibly very large) page as a store would seem problematic. Then again, to have a store for each individual block seems like I might be losing out on a lot of the benefits of SyncedStore.

[tslocke]

@YousefED any comments on this question?

So to handle the entire (possibly very large) page as a store would seem problematic. Then again, to have a store for each individual block seems like I might be losing out on a lot of the benefits of SyncedStore.

[arunbahl]

I'm really interested in a similar use case (smaller database-backed blocks). Out of curiosity, @tslocke, what approach did you end up using?

[tslocke]

UUIDs : ) Or rather - nanoid, which gives you lightly shorter IDs, but same difference. The idea I outlined this issue was a huge PITA. Simplicity wins.