Support toBlob()

[greigs]

toDataURL() works great but the overhead of decoding base64 makes it inefficient for my purposes (rendering to an LED panel). I looked at supporting toBlob() but it uses a callback, unlike toDataURL(), and I can't see how to do this with rdp().

[indus]

Hi @greigs The callback is not the problem. The trick here is to use a Promise (and set "awaitPromise": true) that gets resolved in the remote callback. But you are right that wasn't needed in any other function so it would require some special handeling. I have made some testes and came up with the following.

toBlob: {
        enumerable: true,
        value: function (callback, mimeType, qualityArgument) {

            rdp.ws.once('message', data => {
                data.error && logger.error(data.error);
                !data.error && logger.log(data.result);
                data = JSON.parse(data);
                var err = data.error || data.result.wasThrown ? data.result.result.description : null, res = err ? null : data.result.result;
                if (callback)
                    callback(res);
            });

            rdp.ws.send(`{"id":0,"method":"Runtime.evaluate", "params":{"expression":"${rdp.req.cmd};

                new Promise(resolve => {
                    ${this._uid}.toBlob((blob)=>{ resolve(blob) })
                }, ${mimeType}, ${qualityArgument})

            ", "awaitPromise": true}}`, (err) => { console.error(err); });
        }
    }

(this would be located in the Canvas Proxy implementation at line 386 in ncc.ts or 671 in ncc.js )

But there is still a big problem - the remote debugging protocoll doesn't allow to transfer a blob (for what I know). Everything has to be stringified. So the result you get would be something like:

{
  type: 'object',
  className: 'Blob',
  description: 'Blob',
  objectId: '{"injectedScriptId":1,"id":1}'
}

You could try to create a string from the Blob with something like URL.createObjectURL(blob) or TypedArrays and transfer that but I doubt that this will bring better performance then base64 encoding.

If you find a way to transfer the Blob via RDP with a better performance then base64 encoding, I would be happy to hear from you and add the functionality.

[greigs]

Hah. I had just come to the same conclusion. I tried sending back a byte array as a promise with that flag enabled instead of a blob but it seems strings and json are the only supported callback formats. There is the possibility of sending data as a stream (maybe from a blob), but again it looks like only string data is supported, no binary. https://chromedevtools.github.io/devtools-protocol/tot/IO#method-read

[greigs]

I am very surprised that int arrays don't work. I guess they just don't intend it to be used for data transfer. I was trying this... it recognises the type but I get no data. rdp('new Promise(resolve=>canvas.toBlob(resolve,\'image/jpeg\',0.8)).then(x =>x.arrayBuffer()).then(x => new Uint8Array(x))')

response.. but no way to access the actual data

{
  className:"Uint8Array"
  description:"Uint8Array(7710)"
  objectId:"{"injectedScriptId":1,"id":1}"
  subtype:"typedarray"
  type:"object"
}

[greigs]

Unsurprisingly non-string serialisation isn't really a thing in js, but that doesn't mean base64 is the best way to serialise an image for performance. Looking at things like this, Avro might be one way.... https://github.com/mattheworiordan/nodejs-encoding-benchmarks

[greigs]

Update on this. Base64 is quite efficient to use because it uses speedy native methods atob() and btoa(). Decided to use these going forward. Even with the overhead of these long strings being created, the efficiency of the actual encoding / decoding makes up for this. So using toBlob() isn't essential. Closing for now.