Google Docs will now use canvas based rendering: this may impact some Chrome extensions

[RobertJGabriel]

Just bring it up as and issue and will be willing to help on any develop to get it ready.

Here is the canvas based example https://docs.google.com/document/d/1N1XaAI4ZlCUHNWJBXJUBFjxSTlsD5XctCz6LB3Calcg/preview

@menicosia @ken107 @bboydflo @Amaimersion @JensPLarsen

[jakobsturm]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

[komms]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

How to get whitelisted? What to expect after whitelisting?

[jakobsturm]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

How to get whitelisted? What to expect after whitelisting?

You can get whitelisted here: https://docs.google.com/forms/d/e/1FAIpQLScFxMgvXlq2KMsp0UIM66pvThTF1hpojiXQTqyq9txW79OWag/viewform

Once you are whitelisted you will be able to use your extension ID to convert the canvas to a readable DOM.

Important: Google will not inform you once you are whiltelisted, you need to constantly try it to see whether you are whitelisted yet. Previous answers include IDs that you can use to test it while you wait for your extension to be whitelisted

[komms]

This is not that easy. But here are some hints. If you have doubt please reach out to me through linkedin.com/in/dbjpanda

I am just pasting some hints. So if you copy paste it might not work. Try to understand the snippets.

Basically you need to create a exact overlay of the rect element and put it on top of that rect element to get the cursor point by enabling the pointer event.

    const cursor = document.querySelector('#kix-current-user-cursor-caret');
    const cursorBbox = cursor.getBoundingClientRect();
    const x = Math.floor(cursorBbox.right);
    const y = Math.floor(cursorBbox.top);
    const rect = this.getRect(x, y);

     getRect(x, y) {
        if (!this.styleElement) {
            this.styleElement = document.createElement('style');
            this.styleElement.id = "enable-pointer-events-on-rect";
            this.styleElement.textContent = [
                `.kix-canvas-tile-content{pointer-events:none!important;}`,
                `#kix-current-user-cursor-caret{pointer-events:none!important;}`,
                `.kix-canvas-tile-content svg>g>rect{pointer-events:all!important; stroke-width:7px !important;}`,
            ].join('\n');

            const parent = document.head || document.documentElement;
            if (parent !== null) {
                parent.appendChild(this.styleElement);
            }
        }

        this.styleElement.disabled = false;
        const rect = document.elementFromPoint(x, y);
        this.styleElement.disabled = true;

        return rect;
    }

    getCaretIndex(rect, x, y) {
        const text = rect.getAttribute('aria-label');
        const textNode = document.createTextNode(text);
        const textElement = this.createTextOverlay(rect, text, textNode);

        if (!text || !textElement || !textNode) return null;

        let range = document.createRange();
        let start = 0;
        let end = textNode.nodeValue.length;

        while (end - start > 1) {
            const mid = Math.floor((start + end) / 2);
            range.setStart(textNode, mid);
            range.setEnd(textNode, end);
            const rects = range.getClientRects();
            if (this.isPointInAnyRect(x, y, rects)) {
                start = mid;
            } else {
                if (x > range.getClientRects()[0].right) {
                    start = end;
                } else {
                    end = mid;
                }
            }
        }

        const caretIndex = start;
        textElement.remove();
        return caretIndex;
    }

   createTextOverlay(rect, text, textNode) {
        if (!rect || rect.tagName !== 'rect') return {};

        const textElement = document.createElementNS('http://www.w3.org/2000/svg', 'text');
        const transform = rect.getAttribute('transform') || '';
        const font = rect.getAttribute('data-font-css') || '';

        textElement.setAttribute('x', rect.getAttribute('x'));
        textElement.setAttribute('y', rect.getAttribute('y'));
        textElement.appendChild(textNode);
        textElement.style.setProperty('all', 'initial', 'important');
        textElement.style.setProperty('transform', transform, 'important');
        textElement.style.setProperty('font', font, 'important');
        textElement.style.setProperty('text-anchor', 'start', 'important');

        rect.parentNode.appendChild(textElement);

        const elementRect = rect.getBoundingClientRect();
        const textRect = textElement.getBoundingClientRect();
        const yOffset = ((elementRect.top - textRect.top) + (elementRect.bottom - textRect.bottom)) * 0.5;
        textElement.style.setProperty('transform', `translate(0px,${yOffset}px) ${transform}`, 'important');

        return textElement;
    }

    isPointInAnyRect(x, y, rects) {
        for (const rect of rects) {
            if (x >= Math.floor(rect.left) && x <= Math.floor(rect.right) &&
                y >= Math.floor(rect.top) && y <= Math.floor(rect.bottom)) {
                return true;
            }
        }
        return false;
    }    

is this with whitelisting id or works without it as well?

[RafaOstrovskiy]

This is not that easy. But here are some hints. If you have doubt please reach out to me through linkedin.com/in/dbjpanda I am just pasting some hints. So if you copy paste it might not work. Try to understand the snippets. Basically you need to create a exact overlay of the rect element and put it on top of that rect element to get the cursor point by enabling the pointer event.

    const cursor = document.querySelector('#kix-current-user-cursor-caret');
    const cursorBbox = cursor.getBoundingClientRect();
    const x = Math.floor(cursorBbox.right);
    const y = Math.floor(cursorBbox.top);
    const rect = this.getRect(x, y);

     getRect(x, y) {
        if (!this.styleElement) {
            this.styleElement = document.createElement('style');
            this.styleElement.id = "enable-pointer-events-on-rect";
            this.styleElement.textContent = [
                `.kix-canvas-tile-content{pointer-events:none!important;}`,
                `#kix-current-user-cursor-caret{pointer-events:none!important;}`,
                `.kix-canvas-tile-content svg>g>rect{pointer-events:all!important; stroke-width:7px !important;}`,
            ].join('\n');

            const parent = document.head || document.documentElement;
            if (parent !== null) {
                parent.appendChild(this.styleElement);
            }
        }

        this.styleElement.disabled = false;
        const rect = document.elementFromPoint(x, y);
        this.styleElement.disabled = true;

        return rect;
    }

    getCaretIndex(rect, x, y) {
        const text = rect.getAttribute('aria-label');
        const textNode = document.createTextNode(text);
        const textElement = this.createTextOverlay(rect, text, textNode);

        if (!text || !textElement || !textNode) return null;

        let range = document.createRange();
        let start = 0;
        let end = textNode.nodeValue.length;

        while (end - start > 1) {
            const mid = Math.floor((start + end) / 2);
            range.setStart(textNode, mid);
            range.setEnd(textNode, end);
            const rects = range.getClientRects();
            if (this.isPointInAnyRect(x, y, rects)) {
                start = mid;
            } else {
                if (x > range.getClientRects()[0].right) {
                    start = end;
                } else {
                    end = mid;
                }
            }
        }

        const caretIndex = start;
        textElement.remove();
        return caretIndex;
    }

   createTextOverlay(rect, text, textNode) {
        if (!rect || rect.tagName !== 'rect') return {};

        const textElement = document.createElementNS('http://www.w3.org/2000/svg', 'text');
        const transform = rect.getAttribute('transform') || '';
        const font = rect.getAttribute('data-font-css') || '';

        textElement.setAttribute('x', rect.getAttribute('x'));
        textElement.setAttribute('y', rect.getAttribute('y'));
        textElement.appendChild(textNode);
        textElement.style.setProperty('all', 'initial', 'important');
        textElement.style.setProperty('transform', transform, 'important');
        textElement.style.setProperty('font', font, 'important');
        textElement.style.setProperty('text-anchor', 'start', 'important');

        rect.parentNode.appendChild(textElement);

        const elementRect = rect.getBoundingClientRect();
        const textRect = textElement.getBoundingClientRect();
        const yOffset = ((elementRect.top - textRect.top) + (elementRect.bottom - textRect.bottom)) * 0.5;
        textElement.style.setProperty('transform', `translate(0px,${yOffset}px) ${transform}`, 'important');

        return textElement;
    }

    isPointInAnyRect(x, y, rects) {
        for (const rect of rects) {
            if (x >= Math.floor(rect.left) && x <= Math.floor(rect.right) &&
                y >= Math.floor(rect.top) && y <= Math.floor(rect.bottom)) {
                return true;
            }
        }
        return false;
    }    

is this with whitelisting id or works without it as well?

I guess it won't, bc it operates with svg annotated from canvas after injecting: window._docs_annotate_canvas_by_ext="WHITELISTED_EXTENSION_ID";

[dbjpanda]

I am thinking to create a package https://github.com/dbjpanda/gdocs-utils. I will push some snippets by EOD.

[ElijZhang]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

OK, thanks. (This takes really long ORZ)

[RafaOstrovskiy]

found a way to get the user-selected text or to highlight specific text based on string index.

@ElijZhang Sorry, have you found out a way to get the selected text?

[ElijZhang]

found a way to get the user-selected text or to highlight specific text based on string index.

@ElijZhang Sorry, have you found out a way to get the selected text?

@RafaOstrovskiy I found a not a very good way to get the selected text by analyzing some extension that are in the whitelist.In google docs page which is rendered with _docs_annotate_canvas_by_ext. You may try this code in Chrome devtools:

document.querySelector(".docs-texteventtarget-iframe").contentDocument.execCommand("copy");
const selectedText = document.querySelector(".docs-texteventtarget-iframe").contentDocument.body.innerText

Now you get the selectedText. However, this will change user's clipboard content. I tried to find a way to store user's clipboard content temporarily, and then rewrite it back. But it's hard to do this without the user's awareness. I would be glad If you found a perfect way and tell me the solution.

[swoorpious]

found a way to get the user-selected text or to highlight specific text based on string index.

@ElijZhang Sorry, have you found out a way to get the selected text?

I have spent the last 3 days reverse engineering Language Tool's code, and it's really easy once you get the initial hold of it.

If there is no selection, you will need to make one. You can check this by it's class (.kix-canvas-tile-selection).

To create a selection, you essentially have to simulate mouse events, which then selects the text in docs.

After making a selection, you send a copy event (new CustomEvent("copy")) to the content editable element in the iframe. Google Docs will then put the selected text in that element which you can get by .innerText.

This does not seem to edit clipboard content in my experience.

[tg44]

@swoorpious or @dbjpanda Can you help me with changing text in the doc? Currently I can get the caret, the selected text, I can list the paragraphs, etc. The only thing I really want to do is "typing" into the document. I tried;

// focused element
document.activeElement?.dispatchEvent(new KeyboardEvent("keydown", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keyup", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

// window just in case
window.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

// event target
const b = document.querySelector(".docs-texteventtarget-iframe")?.contentDocument?.body;
  if(b) {
    //console.log(b.querySelector("#docs-texteventtarget-descendant"))
   // with event
    b.querySelector("#docs-texteventtarget-descendant").dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));
   // with forcing
    b.querySelector("#docs-texteventtarget-descendant").innerText = "a";
  }

Seems like none of this worked. What is the trick here?

[luccabb]

@tg44 got write working with the below:

gdocs-enable-annotated-canvas.js

window._docs_annotate_canvas_by_ext = "ogmnaimimemjmbakcfefmnahgdfhfami";

contentScript.js

const writeTextToGDoc = 'hello world'

// write letter to gdoc
function write(letter, doc) {
    const i = new KeyboardEvent("keypress", {
        repeat: !1,
        isComposing: !1,
        bubbles: !0,
        cancelable: !0,
        ctrlKey: !1,
        shiftKey: !1,
        altKey: !1,
        metaKey: !1,
        target: doc,
        currentTarget: doc,
        key: letter,
        code: "Key" + letter.toUpperCase(),
        keyCode: letter.codePointAt(0),
        charCode: letter.codePointAt(0),
        which: letter.codePointAt(0),
        ...{}
    })
    doc.dispatchEvent(i)
}

setTimeout(() => {
    const doc = document.querySelector(".docs-texteventtarget-iframe").contentDocument
    for (const letter of writeTextToGDoc) write(letter, doc)
}, 5000);

manifest.json

...
"content_scripts": [
        {
            "matches": [
                "*://docs.google.com/document/*"
            ],
            "run_at": "document_start",
            "js": [
                "gdocs-enable-annotated-canvas.js"
            ],
            "world": "MAIN"
        },
        {
            "matches": [
                "*://docs.google.com/document/*"
            ],
            "js": [
                "contentScript.js"
            ],
            "all_Frames": false,
            "run_at": "document_end"
        }
    ],
...

[swoorpious]

document.activeElement?.dispatchEvent(new KeyboardEvent("keydown", { key: "a" })); document.activeElement?.dispatchEvent(new KeyboardEvent("keyup", { key: "a" })); document.activeElement?.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

This should be helpful to anyone looking to type in docs with their code.

@tg44 The way docs handles events is a little unintuitive.

If you want to enter text in the document, you use keypreess events, which specify the code of the character you want to type instead of the character itself. Docs uses code of the character to type, and not the character. Additionally, you will need bubbles: true and cancelable: true for docs to type it.

Note: this enters text at the caret's position

Example code for the above:

static InsertChar (char) {
    const IFTarget = GoogleDocsUtils.GetIFEventTarget();

    for (let i = 0; i < char.length; i++) {
        const eventObj = {
            bubbles: true, // important
            cancelable: true, // important
            key: char,
            keyCode: char.charCodeAt(i), // important
            ctrlKey: false,
            shiftKey: false,
        }

        IFTarget.dispatchEvent(new KeyboardEvent("keypress", eventObj))
    }
    return 1;
}

---

If you want to move the caret with keyboard events, or delete characters (backspace and delete on keyboard), you use keydown events. Again, these will require bubbles: true and cancelable: true for docs to register it.

Example code for the above:

static MoveCaretToLeft (n) {
    const IFTarget = GoogleDocsUtils.GetIFEventTarget();
    const eventObj = {
        bubbles: true, // important
        cancelable: true, // important
        code: "ArrowLeft",
        key: "ArrowLeft",
        keyCode: 37, // important
        ctrlKey: false,
        shiftKey: false,
    }

    for (let i = 0; i < n; i++) {
        IFTarget.dispatchEvent(new KeyboardEvent("keydown", eventObj))
    }

    return 1;
}

---

If you want to select text and move it to other locations in the document, you will need a sequence of mouse and keyboard events. You can try to write them based on how you select text with your mouse. Mouse events need to be dispatched at screen coordinates (pixels) that you can calculate with indexes or caret position.

sorry for not replying in time, i was busy with school haha :smile:

[zdw189803631]

@swoorpious or @dbjpanda Can you help me with changing text in the doc? Currently I can get the caret, the selected text, I can list the paragraphs, etc. The only thing I really want to do is "typing" into the document. I tried;

// focused element
document.activeElement?.dispatchEvent(new KeyboardEvent("keydown", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keyup", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

// window just in case
window.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

// event target
const b = document.querySelector(".docs-texteventtarget-iframe")?.contentDocument?.body;
  if(b) {
    //console.log(b.querySelector("#docs-texteventtarget-descendant"))
   // with event
    b.querySelector("#docs-texteventtarget-descendant").dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));
   // with forcing
    b.querySelector("#docs-texteventtarget-descendant").innerText = "a";
  }

Seems like none of this worked. What is the trick here?

Can you tell me how to get the selected text? I looked at the sample code above and I can insert text.

[dbjpanda]

@zdw189803631 You can reach out to me https://www.linkedin.com/in/dbjpanda/ or send me an email, drop your code, I will try to look into it.

[zdw189803631]

@dbjpanda i have send email by https://dbjpanda.me/contact website

[noorhuzaifa]

@zdw189803631 How are you reading the data from docs and did you get a response from @dbjpanda I tried reaching out but the linkedin doesn't exist on the mentioned url.