[Firemonkey] How to access properties of the response object returned by GM.fetch()?

[CrendKing]

console.log(typeof await GM.fetch('https://www.example.com'))

prints "string". However, replacing GM.fetch with normal fetch gives an object with all the properties such as ok and url in it.

So is there a way to for example get the URL or status code of the response object?

[erosman]

The fetch API provides predefined returns as seen on Using Fetch.

GM_fetch has been simplified to return these. There is usually no need to check for the fowling as they become part of the success/failure of the Promise.

• Response.status — An integer (default value 200) containing the response status code.
• Response.statusText — A string (default value ""), which corresponds to the HTTP status code message. Note that HTTP/2 does not support status messages.
• Response.ok — seen in use above, this is a shorthand for checking that status is in the range 200-299 inclusive. This returns a Boolean.

FireMonkey GM_fetch Help:

responseType: (Optional, FireMonkey only) You can set a responseType for the response e.g: 'text' (default), 'json', 'blob', 'arrayBuffer', 'formData'

Therefore await GM.fetch('https://www.example.com') returns the default TEXT.

Here are some examples:

// --- getting text()
const response = await fetch('http://example.com/');
const data = await response.text();

// GM.fetch
const data = await GM.fetch('http://example.com/');

// --- getting json()
const response = await fetch('http://example.com/data.json');
const data = await response.json();

// GM.fetch
const data = await GM.fetch('http://example.com/data.json', {responseType: 'json'}));

// --- getting arrayBuffer()
const response = await fetch('http://example.com/');
const data = await response.arrayBuffer();

// GM.fetch
const data = await GM.fetch('http://example.com/', {responseType: 'arrayBuffer'}));

// --- getting blob()
const response = await fetch('http://example.com/');
const data = await response.blob();

// GM.fetch
const data = await GM.fetch('http://example.com/', {responseType: 'blob'}));

// --- getting formData()
const response = await fetch('http://example.com/');
const data = await response.formData();

// GM.fetch
const data = await GM.fetch('http://example.com/', {responseType: 'formData'}));

I had no feedback on GM.fetch so far. If there is a demand, I can change the GM.fetch to return the same value as normal fetch :thinking:

[CrendKing]

Suppose I GM.fetch() an URL that redirect to a different URL, how do I get that final URL? Basically the fetch version of this:

GM.xmlHttpRequest({
    url: <url>,
    method: 'HEAD',
    onload: response => console.log(response.responseURL)
})

I believe the current GM.fetch() can't do this, right?

The normal fetch can do

fetch(<url>, { method: 'HEAD' })
.then(response => console.log(response.url))

I can change the GM.fetch to return the same value as normal fetch 🤔

Doesn't that break backward compatibility? Maybe a new GM.fetch2() (I hate that name but you might have to).

[erosman]

I can change the GM.fetch to return the same value as normal fetch :thinking:

Not a good idea..... forget it

I am going to update GM.fetch. The current options are mostly suited to 'GET' method. I will make it return the object in case of 'HEAD'. It will be in v2.20 (once I get it working).

[CrendKing]

Frankly, I don't like returning different types of objects based on input value. It not only confuses users as why that happens, but also making you later changing the API more difficult. Also, if HEAD takes special treatment today, but tomorrow someone asks for POST, will you keep doing it?

I know I'm saying this probably because most of the time I program in C/C++. You really can't elegantly do that. But I like static typed language because sometimes I can use an API by just looking at their signature, if the function and parameter names are well chosen.

If I were you, I'd make GM.fetch() as close to fetch() as possible. Because why write several paragraphs of docs and maintain the custom implementation if MDN/Mozilla already did for you? Just deal with the BC issue and call for it.

[erosman]

Also, if HEAD takes special treatment today, but tomorrow someone asks for POST, will you keep doing it?

I have to develop based on circumstances. I hadn't covered the situation that you mentioned with the HEAD so I updated the API accordingly. If we come across another uncovered situation I would try to accommodate (as long as it is possible).

If I were you, I'd make GM.fetch() as close to fetch() as possible. Because why write several paragraphs of docs and maintain the custom implementation if MDN/Mozilla already did for you? Just deal with the BC issue and call for it.

That sounds good... but there are some limitations due to X-ray Vision. It is not possible to return the Response object. Therefore, userscript can not get response.json() or response.blob() or response.arrayBuffer() or response.formData() or response.text() from the Response object. Even with the HEAD requests, I had to rebuild an object (and couldn't pass Response.headers).

The way it is now for v2.20

// returns response object on HEAD request
const response = await GM.fetch('https://example.com/etc', {method: 'HEAD'});

// example of response object on HEAD request
{
  "ok": true,
  "redirected": true,
  "status": 200,
  "statusText": "OK",
  "type": "basic",
  "url": "https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch"
}

[CrendKing]

This should solve my immediate need. Thank you.

[erosman]

v2.20 is out. Let me know if there are any issues to reopen this topic.

[CrendKing]

Working as expected. Thanks!

[AlttiRi]

What GM.fetch are you talking about? Greasemonkey does not have such API: https://wiki.greasespot.net/Greasemonkey_Manual:API TM, VM too.

Based on the example above, why do you name that function fetch? It has not compatible API with window.fetch. It would mislead people. In fact it would makes more sense to name GM.axios.

I see no sense with GM.fetch that has the different API from fetch (That's a some reinventing the wheel) and without the main feature of fetch — a streamable Response if you implement such API as the web extension developer.

GM.fetch that just resolves with entire response looks like just a poor wrapper for the existing GM.xmlhttpRequest.

I usually just download resources so, for me I have written such wrapper that emulates some parts of fetch for my purposes (the implementing of streaming with just wrapping of GM.xmlhttpRequest is not possible, it can be done by the extension's developer only):

The code

```js // Using: const response = await fetch(url); const {status, statusText} = response; const lastModifiedSeconds = response.headers.get("last-modified"); const blob = await response.blob(); // The simplified `fetch` — wrapper for `GM.xmlHttpReques` async function fetch(url) { return new Promise((resolve, reject) => { const blobPromise = new Promise((resolve, reject) => { GM.xmlHttpRequest({ method: "get", url, responseType: "blob", onload: async (response) => resolve(response.response), onerror: reject, onreadystatechange: onHeadersReceived }); }); function onHeadersReceived(response) { const { readyState, responseHeaders, status, statusText } = response; if (readyState === 2) { // HEADERS_RECEIVED const headers = parseHeaders(responseHeaders); resolve({ headers, status, statusText, blob: () => blobPromise }); } } }); } function parseHeaders(headersString) { class Headers { get(key) { return this[key.toLowerCase()]; } } const headers = new Headers(); for (const line of headersString.split("\n")) { const [key, ...valueParts] = line.split(":"); // last-modified: Fri, 21 May 2021 14:46:56 GMT if (key) { headers[key.trim().toLowerCase()] = valueParts.join(":").trim(); } } return headers; } ```

In that code "fetch" correctly resolves on HEADERS_RECEIVED event, and then response resolves on load event.

---

I think that the most correct implementation of GM.fetch should be that I have described here: https://github.com/Tampermonkey/tampermonkey/issues/1278

With streaming and with extending Fetch API.

[erosman]

What GM.fetch are you talking about? Greasemonkey does not have such API: https://wiki.greasespot.net/Greasemonkey_Manual:API TM, VM too.

FireMonkey has a number of new GM APIs. Please refer to the Help for further info.

Based on the example above, why do you name that function fetch? It has not compatible API with window.fetch. It would mislead people. In fact it would makes more sense to name GM.axios.

Since it is based on its window function

window.XMLHttpRequest   -----> GM.XMLHttpRequest 
window.fetch            -----> GM.fetch

GM.XMLHttpRequest is based on XMLHttpRequest but is run from background script. GM.fetch is based on fetch but is run from background script.

I see no sense with GM.fetch that has the different API from fetch

It is there for anyone who wants to use it.

GM.fetch that just resolves with entire response looks like just a poor wrapper for the existing GM.xmlhttpRequest.

GM.fetch/fetch returns a Promise while GM.xmlhttpRequest returns a callback function. Please refer to the Help for more info.

[AlttiRi]

It is not possible to return the Response object.

I think it's possible.

You need to read the chucks of data from the stream in the background script, then transmit them to the content script and then to web script.

const reader = response.body.getReader();
while (true) {
    const {done, value} = await reader.read(); // value is Uint8Array
    sendToContentScript({done, value});
    if (done) {
        break;
    }   
}

In web script you create manually ReadableStream and put the received chunk to it.

new ReadableStream({
    async start(controller) {
        while (true) {
            const {done, value} = await getDataChunkFromBackgroudScript(); // value is Uint8Array
            if (done) {
                break;
            }
            controller.enqueue(value);
        }
        controller.close();
    }
});

With ReadableStream you can easily create Response.

You do the same thing that you do currently, but the difference is that you currently send one big chunk one time on load event, but with streaming you will send tiny chunks multiple times while downloading.

In this case you don't store the downloaded data in the background script, that reduces memory consuming twice.

https://github.com/Tampermonkey/tampermonkey/issues/1050

[erosman]

Let's see if there is a popular demand for it first. Then we shall see what is possible in userScript context (not the same as GM|TM|VM).

[AlttiRi]

Yeah, transmitting of the stream from the background script to the content script is definitely possible as well as creating of Response object.

---

The very simplified demo:

content script:

let resolve;
let promise;
function updatePromise() {
    promise = new Promise(_resolve => {
        resolve = _resolve;
    });
}
updatePromise();

const port = chrome.runtime.connect({name: "demo-fetch"});
port.onMessage.addListener(async function({done, value, i}) {
    // console.log({done, value, i});
    const ab = await fetch(value).then(r => r.arrayBuffer());
    const u8a = new Uint8Array(ab);

    // console.log(i, u8a);
    resolve({done, value: u8a, i});
    updatePromise();
});

const rs = new ReadableStream({
    async start(controller) {
        while (true) {
            const {done, value} = await promise;
            if (done) {
                break;
            }
            controller.enqueue(value);
        }
        controller.close();
    }
});

new Response(rs)
    .blob()
    .then(blob => {
        // console.log(blob);
        const a = document.createElement("a");
        a.href = URL.createObjectURL(blob);
        a.download = "GrimGrossKoodoo.mp4";
        a.click();
        new Promise(resolve => setTimeout(resolve, 1000)).then(() => URL.revokeObjectURL(a.href));
    });

background script:

chrome.runtime.onConnect.addListener(async function(port) {
    console.log(port);
    if (port.name === "demo-fetch") {
        let i = 0;
        const response = await fetch("https://giant.gfycat.com/GrimGrossKoodoo.mp4", {cache: "force-cache"});
        const reader = response.body.getReader();
        while (true) {
            const {done, value} = await reader.read(); // value is Uint8Array
            const blob = new Blob([value]);
            const url = URL.createObjectURL(blob)
            new Promise(resolve => setTimeout(resolve, 1000)).then(() => URL.revokeObjectURL(url));
            port.postMessage({
                done,
                value: url,
                i: i++
            });
            if (done) {
                break;
            }
        }
    }
});

ZIP: https://gist.github.com/AlttiRi/17b22dd2eca503d556e6bac3a6ddf743

[erosman]

As mentioned, let's see the demand.