hackerwins / codepair-old

Real-time markdown editor for interviews, meetings and more...
https://codepair-old.yorkie.dev
Apache License 2.0
90 stars 29 forks source link

Page is stop for more than 1MB text edition. #194

Open Eithea opened 2 years ago

Eithea commented 2 years ago

What happened:

image

I tried to copy and paste 2^20bytes text but the request failed and tried infinitely.

instrument.ts:130 YORKIE E: 
[PP] c:"d3d1899d-b351-44798-c09e-9ceb68d0077" 
err : B {message: 'grpc: received message larger than max (27740248 vs. 4194304)', 
stack: 'Error: grpc: received message larger than max (277…rkie.dev/static/js/2.38ba4ecc.chunk.js:2:2256601)', 
code: 8, metadata: {…}

What you expected to happen:

How to reproduce it (as minimally and precisely as possible): edit over 2^20bytes at once.

Anything else we need to know?:

Environment:

kjw7953 commented 2 years ago

Could I try this issue?!

Eithea commented 2 years ago

@kjw7953 Thank you for your contribution.

https://codepair.yorkie.dev/9340gq There was a data conflict in the CodePair document that you wrote while organizing this issue. To fix the problem, I would like to ask about the editing situation.

Did you see the text crack in your client view while you were writing the document? Or was the text normal while you were editing?

kjw7953 commented 2 years ago

@Eithea Hi, there was no crack in my view when I was writing. I checked that page again two hours after I wrote it, and there was also no problem.

FYI, when I use the codepair for the first time (7.14), the conflict occurred almost immediately.

Eithea commented 2 years ago

@kjw7953 Thank you for your answer.

Considering your answer, the client's local replica editing works normally because of recent bug-fixes. But there seems to be something wrong yet with the process of applying it in the docs of server DB. Your answer was very helpful in finding the cause of the problem. I'll try to fix it quickly.

hackerwins commented 2 years ago

We can consider the following approach:

Approach 1. Splitting the big contents in CodePair

For example:

const bigContent = '...';
const contents = bigContent.splitXXX();
const offset = 0;
for (const content of contents) {
    document.update((root) => {
        root.text.edit(offset, offset, content);
        offset += content.length;
    });
}
  1. CodePair: Split a big content into chunk contents and update them one by one. This will store changes in doc.localChanges.
  2. JS SDK: Add MaxRecvMsgSize options to Client. We can set the value of MaxRecvMsgSize on the server to the client.
  3. JS SDK: The client collects changes from doc.localChanges until the size of changes less than the MaxRecvMsgSize and then sent them when calling client.sync.

If only a part of the change succeeds in sending and the rest fails, there may be a problem that only part of the pasted content remains in the document.