Scaling the json0 type ?

[riston]

Hello, the json0 seems great type for Share.js, although there seems to be some problems with scaling. As the whole structure is based on the JSON and in Node.js/Javascript the JSON.parse, JSON.stringify are not async functions. How to scale these documents, simple new item insert or append requires parsing the whole document, which could be 100Kb, 512Kb, 1MB etc ?

Thank you

[josephg]

Its a good question, and the problem goes all the way down to the database level. Given an operation (say, insert in list) and a document stored in the database, is there a good way to do the right checks, update the document and store it back into the database without having to save & load the whole thing?

For now the answer is to scale out the backend - run multiple sharejs backends. And keep your documents small. In general I don't know what the best answer is. Especially when you are constantly appending to a long list (like sensor data) performance will suffer. What is your use case?

[riston]

Sorry for late reply, yes we are currently running multiple ShareJS (Redis and PostgreSQL) instances. From the design perspective the documents are too large and that causes event loop blocking issues.

Currently the documents are tree structure based, changing one node cause to change change the whole document. When the document size is smaller this approach scales well, but with larger JSON doc's the JSON methods start blocking the event loop:

Single document

{
    "@": {
        points: [ 1,2,3,4 ],
        parent: null
    },

    "a1": {
        points: [ 2,3,4,5 ],
        parent: "@"
    }

    "b1": {
        points: [ 3,4,5 ],
        parent: "@"
    },

    "b1a1": {
        points: [ 2,4,5 ],
        parent: "b1"
    }
}

I haven't tested but one approach to solve the expensive write operation is to split the nodes into separate documents.

[josephg]

Sorry I never got back to you on this. I'd raise this issue in sharedb if you're still struggling with it. The problem is that that code is simply not optimized around large documents. The code assumes documents are much smaller. You're a pioneer exploring what happens with large documents. You'll need to help fix the issues you're running into. Consider making a PR on sharedb!