Content duplication when other users connect.

[binurajrvt]

Description I am building a collaborative editor with TipTap and Hocuspocus with React as shown in the examples in the documentation. After initialisation, I check the number of connected users, and if it is only one (editor.storage.collaborationCursor.users.length), I set the content fetched from a Laravel API (means its an existing document saved in db). Everything works fine till this point. But when a second user logs in, the document content is duplicated across all editor instances. For testing, I am using sqlite driver in hocuspocus server. When other users also connect the content is duplicated just as multiplications of the number of users.

Steps to reproduce the bug Steps to reproduce the behaviour:

1. After initialisation, set the content
2. Login as another user in another browser or incognito.
3. The content gets duplicated on all instances

Expected behaviour Need to set the initial content from a API, if no users have currently opened the document, and it does not get duplicated if other users logs in.

Environment?

• operating system: MAC OS 12.1, Ubuntu
• browser: Chrome, Firefox, Safari
• mobile/desktop: Desktop
• hocuspocus version: 1.0.0-beta.2

[fredpedersen]

Could you explain why you're getting the initial content from an API? That's not really how hocuspocus is designed to work, you can set the inital document if it's never been opened before but after that it should really be persisted in binary form and retreived onloaddocument.

This thread might be of use #344

[binurajrvt]

Thank you for the response. I am saving the content as html in MySQL db and from there we convert it to XML and supply for another projects as response to an API request from that project. We need the HTML format. "set the inital document if it's never been opened before" - is there any method to check if the document is not opened? That means the initial content in binary format is to be in Hocuspocus SQLite/RocksDB only or should I save the content in binary format (with Y.encodeStateAsUpdate(ydoc)) also in the API MYSQL database where I save html format content also and fetch that data and set the initial content with Y.applyUpdate(ydoc, props.savedYDoc) as seen somewhere. Kindly advice. Any code samples will be of great help.

[fredpedersen]

You need to be saving it as a UInt8Array somewhere and using that as the primary source for the hocus server. It depends on your app setup but it's probably best achieved by using the database extension and storing the document as per the example code on the docs.

To get the document as html you have two options, again depending on your setup:

1. You can additionally store it as html during the database extensions store callback by converting the ydoc given as a parameter to your format. @hocuspocus/transformers may be useful here
2. The other option is when you need to get the html/xml you can fetch the data in yjs format, and parse it there to html - you'll need to do this on nodejs or similar as yjs only has JS support at the moment

My preference would be for options two because:

1. You don't end up duplicating any data
2. There will be many calls to database store function every time someone updates the document, so doing extra work could slow the server down

[binurajrvt]

Thank you again for the quick response. I used editor.getHTML(); as given in TipTap examples to fetch the html. I will try with as you suggested that and will post here. Again:

"set the inital document if it's never been opened before"

How to make sure if its never opened?

[fredpedersen]

If it's been opened, it will exist in your table as yjs doc via the database extension

[janthurau]

@binurajrvt let us know if you're still experiencing issues, but if you keep using the ydoc it should work fine :-) Closing for now.

[binurajrvt]

Thank you so much for the suggestions. I am still working on it.

[binurajrvt]

Currently, being the development/learning phase, the Hocuspocus server code is kept minimum as shown in the examples:

import { SQLite } from "@hocuspocus/extension-sqlite";
const server = Server.configure({
    port: 1235,
    extensions: [new SQLite({ database: "c_db_01" })],
});

server.listen();

And the client side React code, first I want to check whether the document is already created or not. And I fetch the already saved content from my Laravel API (Not hocuspocus db). This is saved in the API with a save button click from the UI:

const handleSave = () => {
    const txtHTML = editor.getHTML();
    const saveDoc = fromUint8Array(Y.encodeStateAsUpdate(ydoc));
    const endPoint = process.env.API_BACKEND_BASENAME + "/content";
    const postData = {
        content: txtHTML,
        email: authContext.email,
        document_id: localStorage.getItem("document_id"),
        ydoc_content: saveDoc,
    };

    axios.post(endPoint, postData).then((response) => {
        console.log(response);
    });
};

And as written above, first I fetch this content from the Laravel API and set the Initial content

const ydoc = new Y.Doc();
...
...

Y.applyUpdate(ydoc, toUint8Array(savedYDoc));

...
...

Rest of the code in the UI to initialise the editor, is followed exactly as given in the the examples.

Earlier the content duplication issue happened when I used:

editor.commands.setContent(props.content);

Now, changed that and after using Y.applyUpdate(ydoc, toUint8Array(savedYDoc)); the content duplication issue does not happen. This was inspired from this post: https://github.com/yjs/yjs-demos/issues/16#issuecomment-614184336 Now I would like to know:

1. Whether this is the correct way of doing it.
2. From react client side, how to check if document is already opened, before initialising const ydoc = new Y.Doc();.
3. Should I set anything else in hocuspocus server code.

Code sample will be highly appreciated. Thanking in advance

[fredpedersen]

It seems like there are a few points of confusion here, perhaps the documentation needs some bits adding.

You do not need to set the content of the editor, you can use the collaboration plugin which will handle this for you. The collaboration plugin links a tiptap editor, to a Hocuspocus provider. The provider handles the connection to hocuspocus server and should be the only thing handling data in/out. You shouldn't be fetching the document client side using react code - again the provider linked with the collaboration plugin will do all this for you.

Hocuspocus server will store the document in-memory while it's open, plus it will store the document in the database as a Y.Doc binary for when the connection is closed for retrieval later. This is all you need to get everything running. The server will get the already opened documents from the database if anyone connects and tries to open them.

You can additionally add hooks when for when the document is saved/changed ect depending on your use case to make other things happen. It sounds like you might want the webhook extension to let laravel know when the document is saved, so you can also save it in a laravel compatible format such as JSON. There is no need to fetch the document from laravel, unless you have some sort of predefined template format you want the Y.Doc in when its first created. It will be fetched from the database/SQLite extension.

Just to note, I'm not a developer of this package so I'm not an expert, but this is my understanding.

[binurajrvt]

Thank you for giving an excellent insight. So the database/webhook extensions of the Hocuspocus are to be used for data persistence and not in API. I am running front end, server and laravel app in Docker. So I think I need to set a volume for hocuspocus database

[fredpedersen]

Sort of, yes. Essentially you need primary storage of the binary y.doc, most easily achieved using the database extension. The webhook can be used to connect to a laravel API to make things happen, or for secondary storage in another format such as JSON. Secondary storage should not be used to retrieve saved documents, they will come from the primary storage.

Technically the webhook could be used to store the binary and therefore be primary storage but this is probably more complicated than necessary when we have the excellent database extension.

[binurajrvt]

Thank you. Will do as per your suggestion. Also, the database and its schema has to be set up manually (Sqlite or MySQL or RocksDB or any real time db ) and pass the db details to hocuspocus via Server.configure ? Is it possible to get full Hocuspocus server.js code using any such database, as the code in the documentation seems to be incomplete: https://tiptap.dev/hocuspocus/api/extensions/database

[fredpedersen]

You can setup the table using Laravels migrations system something like this:

return new class extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('documents', function (Blueprint $table) {
            $table->id();
            $table->binary('data');
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('documents');
    }
};

In Server.mjs I'd recommend using dotenv package to get the same .env information Laravel uses and a mysql pool for connections, for example:

import mysql from 'mysql2';
import dotenv from 'dotenv';

dotenv.config()

const pool = mysql.createPool({
    connectionLimit: 100, //important
    host: '127.0.0.1',
    user: process.env.DB_USERNAME,
    password: process.env.DB_PASSWORD,
    database: process.env.DB_DATABASE,
    debug: false
});

You can then write your sql statements in the database extension using pool.query.

[binurajrvt]

Thank you so much for the this answer along with code that gave a better understanding of how it works. If the documentation at Hocuspocus site, had specified that the db should be created externally, it would have avoided the confusion. I will give it a try and will let you know.

[binurajrvt]

How this mysql connection and query to be used when using the database extension @hocuspocus/extension-database as per the example shown in: https://tiptap.dev/hocuspocus/api/extensions/database ? If the server code was complete it would be of so much help.

[binurajrvt]

Is it possible to get the full server side code along with db connection pool based on this code featured on hocuspocus site?

import { Server } from '@hocuspocus/server'
import { Database } from '@hocuspocus/extension-database'
import sqlite3 from 'sqlite3'

const server = Server.configure({
  extensions: [
    new Database({
      // Return a Promise to retrieve data …
      fetch: async ({ documentName }) => {
        return new Promise((resolve, reject) => {
          this.db?.get(`
            SELECT data FROM "documents" WHERE name = $name ORDER BY rowid DESC
          `, {
            $name: documentName,
          }, (error, row) => {
            if (error) {
              reject(error)
            }

            resolve(row?.data)
          })
        })
      },
      // … and a Promise to store data:
      store: async ({ documentName, state }) => {
        this.db?.run(`
          INSERT INTO "documents" ("name", "data") VALUES ($name, $data)
            ON CONFLICT(name) DO UPDATE SET data = $data
        `, {
          $name: documentName,
          $data: state,
        })
      },
    }),
  ],
})

server.listen()

[fredpedersen]

That's a bit outside the scope of hocus-pocus. You can find out how to use the mysql2 node js library from their documentation. Essentially just replace this.db?.get with pool.query from memory. Passing values may be slightly different too. It's all well documented.

On Fri, 17 Feb 2023, 12:55 binurajrvt, @.***> wrote:

Is it possible to get the full server side code along with db connection pool based on this code featured on hocuspocus site?

import { Server } from @.***/server'

import { Database } from @.***/extension-database'

import sqlite3 from 'sqlite3'

const server = Server.configure({

extensions: [

new Database({

  // Return a Promise to retrieve data …

  fetch: async ({ documentName }) => {

    return new Promise((resolve, reject) => {

      this.db?.get(`

        SELECT data FROM "documents" WHERE name = $name ORDER BY rowid DESC

      `, {

        $name: documentName,

      }, (error, row) => {

        if (error) {

          reject(error)

        }

        resolve(row?.data)

      })

    })

  },

  // … and a Promise to store data:

  store: async ({ documentName, state }) => {

    this.db?.run(`

      INSERT INTO "documents" ("name", "data") VALUES ($name, $data)

        ON CONFLICT(name) DO UPDATE SET data = $data

    `, {

      $name: documentName,

      $data: state,

    })

  },

}),

],

})

server.listen()

— Reply to this email directly, view it on GitHub https://github.com/ueberdosis/hocuspocus/issues/509#issuecomment-1434470250, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6COEL6O5TZRIYDWOMM6U3WX5KJPANCNFSM6AAAAAAUSJODFM . You are receiving this because you commented.Message ID: @.***>

[eddiej]

Could you explain why you're getting the initial content from an API? That's not really how hocuspocus is designed to work, you can set the inital document if it's never been opened before but after that it should really be persisted in binary form and retreived onloaddocument.

This thread might be of use #344

What's the correct way to set the initial content? I think this is the exact part that's causing confusion here (and with me also). Often times, the initial contents of an editable field might have been created by a script or pulled from somewhere else, this is before the content is ever displayed to a user via an interface. Where / how exactly should initial content like this be set?

[binurajrvt]

Thanks to your help, I managed to store the content in persistent databases. Tested in FireBase and also in MySQL running in Docker with persistent volume, and now RethinkDB realtime db. But the problem with the entire project is that, if you type fast multiple lines, the collaboration fails even though the cursor of other users are correctly shown. The error on console is:

sync.js:86 Caught error while handling a Yjs update RangeError: Maximum call stack size exceeded

What could be the reason? When this error happens, the collaboration not working, means it wont sync document changes, but other users cursors are shown correctly. Using some custom extensions like Comments etc. package.json dependencies are:

    "dependencies": {
        "@hocuspocus/extension-sqlite": "^1.0.0-beta.6",
        "@hocuspocus/provider": "^1.0.0-beta.2",
        "@hocuspocus/server": "^1.0.0-beta.2",
        "@react-keycloak/web": "^3.4.0",
        "@testing-library/jest-dom": "^5.16.5",
        "@testing-library/react": "^13.4.0",
        "@testing-library/user-event": "^13.5.0",
        "@tiptap/extension-blockquote": "^2.0.0-beta.199",
        "@tiptap/extension-character-count": "^2.0.0-beta.199",
        "@tiptap/extension-code-block-lowlight": "^2.0.0-beta.199",
        "@tiptap/extension-collaboration": "^2.0.0-beta.199",
        "@tiptap/extension-collaboration-cursor": "^2.0.0-beta.199",
        "@tiptap/extension-color": "^2.0.0-beta.199",
        "@tiptap/extension-highlight": "^2.0.0-beta.199",
        "@tiptap/extension-paragraph": "^2.0.0-beta.199",
        "@tiptap/extension-table": "^2.0.0-beta.199",
        "@tiptap/extension-table-cell": "^2.0.0-beta.199",
        "@tiptap/extension-table-header": "^2.0.0-beta.199",
        "@tiptap/extension-table-row": "^2.0.0-beta.199",
        "@tiptap/extension-task-item": "^2.0.0-beta.199",
        "@tiptap/extension-task-list": "^2.0.0-beta.199",
        "@tiptap/extension-text-align": "^2.0.0-beta.199",
        "@tiptap/extension-text-style": "^2.0.0-beta.199",
        "@tiptap/react": "^2.0.0-beta.113",
        "@tiptap/starter-kit": "^2.0.0-beta.189",
        "@types/google.accounts": "^0.0.2",
        "axios": "^1.1.3",
        "jstoxml": "^3.2.5",
        "jwt-decode": "^3.1.2",
        "jwt-encode": "^1.0.1",
        "keycloak-js": "^17.0.0",
        "lowlight": "^2.7.0",
        "react": "^18.2.0",
        "react-bootstrap": "^2.7.0",
        "react-dom": "^18.2.0",
        "react-router-dom": "^6.4.3",
        "react-scripts": "^5.0.1",
        "remixicon": "^2.5.0",
        "sass": "^1.55.0",
        "web-vitals": "^2.1.4",
        "yjs": "^13.5.41"
    },

Please help.

[binurajrvt]

Thankfully, the above issue has been sorted out, as it happened from front end with conflicting node packages. Now the problem is if we refresh the page (Not always), when only one user instance is editing the document, the database extension does not set the already edited content. As far as I know, the database extension should fix the initial content which is being edited, even if the page is refreshed. Please correct, if I am wrong. The server.js code is as follows (with MySQL. I already have tried with FireBase, SQLite with both in memory and disk write and RethinkDB):

import * as dotenv from "dotenv";
dotenv.config();

import { Server } from "@hocuspocus/server";
import { Database } from "@hocuspocus/extension-database";
import mysql from "mysql2";
/* 

Should we convert it to Uint8Array or the DB extension doesit by default?

*/

//import { toUint8Array, fromUint8Array } from "js-base64";
import { Throttle } from "@hocuspocus/extension-throttle";

const pool = mysql.createPool({
    connectionLimit: 100,
    host: process.env.DB_HOST,
    user: process.env.DB_USERNAME,
    password: process.env.DB_PASSWORD,
    database: process.env.DB_DATABASE,
    port: process.env.DB_PORT,
    debug: false,
});

const server = Server.configure({
    port: process.env.HOCUSPOCUS_PORT,
    extensions: [
        new Throttle({
            throttle: 200,
            banTime: 1,
        }),

        new Database({
            fetch: async ({ documentName }) => {
                return new Promise((resolve, reject) => {
                    console.log("Trying to fetch");
                    pool?.query(
                        "SELECT data FROM ydocuments WHERE name = ? ORDER BY id DESC",
                        [documentName],
                        (error, row) => {
                            if (error) {
                                reject(error);
                            }
                            console.log("data: " + JSON.stringify(row));
                            if (row && row.data) {
                                resolve(row.data);
                            } else {
                                resolve(null);
                            }
                        }
                    );
                });
            },

            store: async ({ documentName, state }) => {
                pool?.query(
                    "INSERT INTO ydocuments (name, data) VALUES (?, ?) ON DUPLICATE KEY UPDATE data = ?",
                    [documentName, state, state],
                    (error, result) => {
                        if (error) {
                            throw error;
                        }
                        console.log(
                            `inserted/updated ${result.affectedRows}`
                        );
                    }
                );
            },
        }),
    ],
});

server.listen();

Also, should we convert it to Uint8Array or the DB extension doesit by default as commented in the code?