qsniyg / maxurl

Finds larger/original versions of images and videos
https://qsniyg.github.io/maxurl/
Apache License 2.0
1.14k stars 72 forks source link

Popup not working on instagram #719

Open LeGiTiM opened 3 years ago

LeGiTiM commented 3 years ago

Priority: not urgent. The issue is resolvable by using the Developer Tools. It's just annoying.

Hi there! I tested it on Brave, Opera, Firefox Nightly, Firefox ESR: the popup shows the loading cursor, then it displays nothing. They must have changed something which prevents Image Max URL to grab the file. Or is it just me? I'm surprised no one reported this already ^^"

Browsers based on Chromium can still popup videos, strangely, while Gecko browsers cannot. Not sure if this detail can help.

As always, thanks for your work! <3

qsniyg commented 3 years ago

Unfortunately I can't test Instagram for the moment, can you share a screenshot of the console when you open the popup? There should be an error of sorts.

If you wish, we can discuss over discord/matrix (links are in the readme), might be easier as it might require a bit of back&forth :)

LeGiTiM commented 3 years ago

Firefox ESR: FirefoxESR-2021 04 30-1494x1440-44 Brave: BRave screenshot-brave-2021 04 30-1494x1440-45 Opera: Opera screenshot-opera-2021 04 30-1494x1440-46 Firefox Nightly: Firefox Nightly screenshot-firefox-2021 04 30-1494x1440-47 Firefox Nightly 2 screenshot-firefox-2021 04 30-1494x1440-48

qsniyg commented 3 years ago

Wow that's strange. For the moment, you should be able to work around it with one of the following:

Could you send the raw cdninstagram.com URL and the response from the Network tab? Apparently this is due to some cross-origin problem, it might be a temporary error on Instagram's part, or perhaps the script is doing something wrong... I'd like to fix this if possible though :)

(Don't worry about the logging_client_events and ajax/bz ones, those are just tracking URLs)

LeGiTiM commented 3 years ago

Your workaround works (only open in new tab. "partially loaded Streams" does not), it's less of a hassle, while waiting for the popup. Thanks! The cdn image that i'm trying to load: https://scontent-cdg2-1.cdninstagram.com/v/t51.2885-15/e35/s1080x1080/179121655_372708597364355_280138325602191742_n.jpg?tp=1&_nc_ht=scontent-cdg2-1.cdninstagram.com&_nc_cat=1&_nc_ohc=d1kMbXwAtjIAX8FMpSn&edm=AP_V10EBAAAA&ccb=7-4&oh=18a803646681155a0b3e48e870f9a2e2&oe=60B101B8&_nc_sid=4f375e

The network tabs, a few seconds after I triggered the popup to no avail (the loading cursor is gone, and no popup):

ghost commented 3 years ago

I have not noticed such errors in last Firefox Nightly. What is url of the post. You just gave direct url of the image which is useless. I can test it.

LeGiTiM commented 3 years ago

Ok, so after disabling all extension and testing about 20 of them on Firefox then Brave, I found a couple of extensions that may interfere. Weirdly, the issue can be bypassed by disabling certains extensions or settings related to referer policies. The problem is that they are not the same for each browser, which made me unable to identify this in my first tests before reporting. And that everything was working fine a week ago. On Firefox, I was able to restore the popup functionnality by disabling the option "Block tracking headers" in Privacy Possum (while this extension poses no problem in chromium browsers), and in Chromium browsers, I was able by disabling Referer Privacy.

I don't think this issue is solvable on your side: it's facebook who introduced brand new nasty/fishy headers in their instagram bullshit or something like that.

Your script is working fine, as long as we allow these new headers to track us. Sorry for the invonvenience!

qsniyg commented 3 years ago

@LeGiTiM See if https://github.com/qsniyg/maxurl/commit/248f1d89d2a4adde663ad486eaba67a42fb0b7ab fixes it. Sorry, still can't test IG so I'm not sure if it'll work, but I think it should (I tested it with curl and the example url you sent).

Seeing your new comment: It should be possible for the script to support it without needing to disable those extensions. If IG works, then the script should work too, when running on IG. If it doesn't, that's a bug I have to fix.. which has hopefully been fixed with the commit above :)

@AdClear247 If it works for you, then it might be some kind of regional test they're doing. This is definitely intentional on their end though, try this:

$ curl 'https://scontent-cdg2-1.cdninstagram.com/v/t51.2885-15/e35/s1080x1080/179121655_372708597364355_280138325602191742_n.jpg?tp=1&_nc_ht=scontent-cdg2-1.cdninstagram.com&_nc_cat=1&_nc_ohc=d1kMbXwAtjIAX8FMpSn&edm=AP_V10EBAAAA&ccb=7-4&oh=18a803646681155a0b3e48e870f9a2e2&oe=60B101B8&_nc_sid=4f375e'  -H 'authority: scontent-cdg2-1.cdninstagram.com' -H 'pragma: no-cache' -H 'cache-control: no-cache'  -H 'upgrade-insecure-requests: 1'   -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' -H 'sec-gpc: 1' -H 'sec-fetch-site: cross-site' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1'   -H 'sec-fetch-dest: document' -H 'accept-language: en-US,en;q=0.9' -I --compressed
HTTP/2 200 
last-modified: ...
x-haystack-needlechecksum: ...
x-needle-checksum: ...
content-type: image/jpeg
x-fb-server-cluster-forwarded: ...
x-fb-config-version-olb-prod: ...
timing-allow-origin: *
cache-control: max-age=1209600, no-transform
content-length: 66089
x-fb-trip-id: ...
date: ...
cross-origin-resource-policy: same-origin
alt-svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600

$ curl 'https://scontent-cdg2-1.cdninstagram.com/v/t51.2885-15/e35/s1080x1080/179121655_372708597364355_280138325602191742_n.jpg?tp=1&_nc_ht=scontent-cdg2-1.cdninstagram.com&_nc_cat=1&_nc_ohc=d1kMbXwAtjIAX8FMpSn&edm=AP_V10EBAAAA&ccb=7-4&oh=18a803646681155a0b3e48e870f9a2e2&oe=60B101B8&_nc_sid=4f375e'   -H 'authority: scontent-cdg2-1.cdninstagram.com' -H 'pragma: no-cache' -H 'cache-control: no-cache'   -H 'upgrade-insecure-requests: 1'   -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' -H 'sec-gpc: 1' -H 'sec-fetch-site: cross-site' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1'   -H 'sec-fetch-dest: document' -H 'accept-language: en-US,en;q=0.9' -H 'origin: https://instagram.com' -I --compressed
HTTP/2 200 
last-modified: ...
x-haystack-needlechecksum: ...
x-needle-checksum: ...
content-type: image/jpeg
x-fb-server-cluster-forwarded: ...
x-fb-config-version-olb-prod: ...
timing-allow-origin: *
cross-origin-resource-policy: cross-origin
access-control-allow-origin: *     <------- this header is new, only added because of origin: https://instagram.com
cache-control: max-age=1209600, no-transform
content-length: 66089
x-fb-trip-id: ...
date: ...
alt-svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600

Other origins don't add the access-control-allow-origin: * header.

ghost commented 3 years ago

Now I understand. This bug only appears on any Instagram image when popup action is popup in extension. And when uMatrix addon is used with spoof referrer header option active, for www.instagram.com or all sites. Then your addon does not work, popup not displayed. Meaning, this is not fixed yet. As soon as I disable that option in uMatrix, and refresh page, popup now works. Looks like has nothing to do with access-control-allow-origin: * because that response header is present in both cases in your extension's network requests (about:devtools-toolbox?id=maxurl%40qsniyg&type=extension). I even tried copying as curl (windows) both requests (when working popup and when not) and there are no differences. Also response headers look same. The only difference I found is in console.

When not working popup:

Unable to find request ID: ko5pcw8m1hpdp31b background.js:1016:12
    extension_message_handler moz-extension://78d7bd8d-8e03-453f-a476-a7b8606ee893/extension/background.js:1016

When working popup:

Some cookies are misusing the “SameSite“ attribute, so it won’t work as expected 6
Cookie “csrftoken” has “SameSite” policy set to “Lax” because it is missing a “SameSite” attribute, and “SameSite=Lax” is the default value for this attribute. info
Cookie “rur” has “SameSite” policy set to “Lax” because it is missing a “SameSite” attribute, and “SameSite=Lax” is the default value for this attribute. info
Cookie “ds_user_id” has “SameSite” policy set to “Lax” because it is missing a “SameSite” attribute, and “SameSite=Lax” is the default value for this attribute. info
Cookie “csrftoken” has “SameSite” policy set to “Lax” because it is missing a “SameSite” attribute, and “SameSite=Lax” is the default value for this attribute. info
Cookie “rur” has “SameSite” policy set to “Lax” because it is missing a “SameSite” attribute, and “SameSite=Lax” is the default value for this attribute. info
Cookie “ds_user_id” has “SameSite” policy set to “Lax” because it is missing a “SameSite” attribute, and “SameSite=Lax” is the default value for this attribute. info
Unable to find request ID: ko5pl1331kst77lw background.js:1016:12
    extension_message_handler moz-extension://78d7bd8d-8e03-453f-a476-a7b8606ee893/extension/background.js:1016

Still don't know what is the cause. So looks like has nothing to do with network requests? Maybe cookies? I cleared console between retries. As author of post said, problem disappears when such referrer modification is disabled in extensions like uMatrix. Also note that such extensions work by modifying referrer to be same as url of each request.

qsniyg commented 3 years ago

@AdClear247 Thanks for your research into this, really appreciate it!

Note that such extensions work by modifying referrer to be same as url of each request

Then that's the issue. If you specify a referer/origin that isn't instagram.com (even cdninstagram.com will fail), it will not add the access-control-allow-origin: * header, causing it to fail loading the image under the browser's context (more info below). However, what's strange in @LeGiTiM 's case is that Instagram can even work at all with those extensions enabled (you may need to press shift+ctrl+r to clear the cache&reload if it seems to work for you with uMatrix). Perhaps they have some kind of exception for Instagram... but then why would the extension fail?

I guess the extension could also just inject access-control-allow-origin: * into the response headers of images/videos it loads, but this feels like a bit of a hack (it also won't work for the userscript as they can't modify things like that). I could perhaps add an option for it though. Turns out I had already added that. However, it only gets added when Load media anonymously is enabled.

Adding an option to always add the header isn't really possible, because access-control-allow-origin: * will result in an error if credentials are sent (which is not unlikely).

For the moment, Load media anonymously might be the only workaround for this issue (other than, of course, disabling the extensions that change the referer header).

because that response header is present in both cases in your extension's network requests

This shouldn't affect anything though, as any request done within the extension's context will not be affected by access-control-allow-origin. However, that header will affect what can be loaded and displayed within the browser's context (if you open the network tab on the instagram.com page, rather than the extension).

qsniyg commented 3 years ago

https://github.com/qsniyg/maxurl/commit/3ff3bbfea7027109f47dd66ae24a098f7d944345 should fix it for the extension, but it likely won't fix it for the userscript, as it cannot override the access-control-allow-origin header.