search

uBlockOrigin / uMatrix-issues

This is the community-maintained issue tracker for uMatrix
https://github.com/gorhill/uMatrix
123 stars 17 forks source link

File Save As... fails in Firefox (but succeeds when uMatrix is disabled) #92

Open bhrgunatha opened 5 years ago

bhrgunatha commented 5 years ago

Prerequisites

Description

A specific URL where the issue occurs

https://pastebin.com/HjWANtHk

Steps to Reproduce

  1. Visit some site - e.g. pastebin above but it also happens when trying to save json from reddit.
  2. Click raw - the text only file appears.
  3. File | Save Page As... (or CTRL-S)
  4. Choose any location.
  5. Click Save.
  6. Go to Downloads.
  7. See file has Failed.
  8. Click the retry icon (right hand side of downloads list).
  9. File saves correctly.

Similar example

  1. Visit my saved links on reddit - https://www.reddit.com/user/bhrgunatha/saved#links

  2. Change the url suffix saved#links -> saved.json#links to receive json instead of html - https://www.reddit.com/user/bhrgunatha/saved.json#links

  3. File Save As...

  4. Steps same as above.

Ruleset

Active rules

Supporting evidence

Logger output

Your environment

uBlock-user commented 5 years ago

Other column is blocked, the request is tabless and identified in other cell column which is blocked.

gorhill commented 5 years ago

I can reproduce the page not saving, but I cannot reproduce an actual issue. The logger shows a behind-the-scene network request being blocked, hence it's a user ruleset issue, not a uMatrix issue. With the new UI in 1.3.15rc1, I can set a rule pastebin.com 1st-party other allow from the logger and saving then work as expected.

bhrgunatha commented 5 years ago

My global rules: 

https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: edu.swi-prolog.org true
matrix-off: localhost true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: wyciwyg-scheme true
referrer-spoof: behind-the-scene false
* * css allow
* * image allow
* cdnjs.cloudflare.com script allow
* code.jquery.com script allow
* d3js.org script allow
* gcs-vimeo.akamaized.net media allow
* googlevideo.com media allow
* player.vimeo.com frame allow
* player.vimeo.com other allow
* player.vimeo.com script allow
* vimeocdn.com frame allow
* vimeocdn.com media allow
* vimeocdn.com other allow
* vimeocdn.com script allow
* www.youtube.com script allow
* youtube.com frame allow

I don't have any rules for pastebin.

Are you saying in order to use the browser's built in File Save As with umatrix 'other' must be allowed either for specific sites or globally?

Sorry I don't understand where the behind the scene blocking is coming from in the logger - only those 2 cookies.

gorhill commented 5 years ago

In the logger, select "All" or "Behind the scene" and see if you can see behind-the-scene entries.

In any case, the stable version won't allow you to create a rule from that entry in the logger, this has been addressed in 1.3.15rc1, which I should release hopefully soon (but now I need to revert the migration to Fontawesome 5, plus other stuff...)

uBlock-user commented 5 years ago

Add matrix-off: behind-the-scene true and try to reproduce again.

gorhill commented 5 years ago

@uBlock-user It's not going to work, the context is pastebin.com not behind-the-scene. This is a tabless network request, but with a valid context. This is what is fixed in 1.13.15rc1, you can set rules in the proper context instead of the behind-the-scene one.

bhrgunatha commented 5 years ago

Both All and behind-the-scene are blank in the logger - also after adding the matrix-off rule.

However I now see a 1 in the 'other' column - and if I allow that I can save (as usual.)

I don't understand what 'other' really represents though. Can websites use/abuse 'other' network requests?

uBlock-user commented 5 years ago

Can it still be a tabless request if it has a valid context ? I always understood that if it is BTS, it doesn't have a valid context.

uBlock-user commented 5 years ago

I now see a 1 in the 'other' column - and if I allow that I can save (as usual.)

you should whitelist other column for first-party domains.

I don't understand what 'other' really represents though.

Other represents what is not classified under scripts,images,XHR,css, media, cookie, frame etc..

gorhill commented 5 years ago

I don't understand what 'other' really represents though.

https://github.com/gorhill/uMatrix/wiki/The-popup-panel#the-type-cells

gorhill commented 5 years ago

Can it still be a tabless request if it has a valid context ?

Yes. Originally behind-the-scene meant no tab/no context. With Chromium 63 and Firefox Webext, it's possible a network request is not associated with a tab (extenstions framework says tabId === -1), but now it can have context (initiator for Chromium, documentUrl for Firefox) even though there is still no tab. Hence now I use "tab-less" and context-less. The later should be quite rare now, but if ever it occurs, it will be filtered with the context behind-the-scene, that is the only time that special context is used now.

gorhill commented 5 years ago

What's to address?

All worked as expected on my side, the logger reported everything, just a matter of selecting All to see everything. The only quirk is to set rule on a tab-less request and this was fixed in https://github.com/gorhill/uMatrix/issues/951#issuecomment-425634875.

gwarser commented 5 years ago

Something wrong with Firefox. It insists to re-download already visible content when saving, download fails because is blocked, then succeeds on retry. Extensions logger does not show this retry attempt, but it is visible in browser console.

Share your thoughts here: https://bugzilla.mozilla.org/show_bug.cgi?id=1343466

uBlock-user commented 5 years ago

What's to address?

So I will close it then, you kept it open and didn't close it, so I thought you had to address something.

uBlock-user commented 5 years ago

I could only reproduce the second example of reddit link, which is now forbidden, but didn't happen after I whitelisted tabless request.

gorhill commented 5 years ago

Share your thoughts here: https://bugzilla.mozilla.org/show_bug.cgi?id=1343466

Seems to be a UI thing in Firefox. The page was properly saved on my side, except for the blocked items. So it seems Firefox is marking the download as failed just because some items were blocked by uBO:

a

d-g commented 5 years ago

Seems to be a UI thing in Firefox. The page was properly saved on my side, except for the blocked items. So it seems Firefox is marking the download as failed...

As for me, no, the download actually fails. I’ve just encountered this bug using µMatrix 1.3.17b1 after updating Firefox from 60.4.0 ESR to 64.0.2 (current release), so this is clearly a regression.

With FF60 saving works well with µMatrix disabled ‘behind-the-scene’ per default. In FF64 when trying to save any page I only get ‘Failed’ and a bunch of lines like these on the logger console:

14:54:21    www.gnu.org --  other   https://www.gnu.org/graphics/fs-gang.png
14:54:21    www.gnu.org --  other   https://www.gnu.org/print.min.css
...

None of these files are saved despite that they are loaded from the net and shown on a page: I only got *.html and empty *_files directory.

Rules are following:

https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: behind-the-scene true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: vivaldi-scheme true
matrix-off: wyciwyg-scheme true
noscript-spoof: * true
referrer-spoof: * true
referrer-spoof: behind-the-scene false
* * * block
* * css allow
* * frame allow
* * image allow

All other settings are default.

As far as I understood, the only thing could be done is to allow ‘other’ globally, i. e.:

* * other allow

Is it indeed so? If yes, this ‘other’ switch becomes nearly unusable in new Firefox versions, as saving pages is crucial feature of a browser.

What's to address? All worked as expected on my side

In the light of the above, it would be very nice of you, if users were at least prominently notified about such breaking changes even when things have been broken by Mozilla.

gorhill commented 5 years ago

I only get ‘Failed’ and a bunch of lines like these on the logger console

So you got your answer, you need to allow other for www.gnu.org -- you can do this from the logger.

This is on the front page of the uMatrix project (emphasis in original):

Using uMatrix logger is key to un-break sites: the logger will show you all that uMatrix does internally.

gorhill commented 5 years ago

Alright, I am going to give another look at what can be done, if anything. The tablessness issue has always been a thorny one, the is no obvious correct solution.

In the previous version, uMatrix was attempting to find a tab matching the contextual URL of the request, and if found would use this tab to report the network request. This was put in place strictly to provide a temporary mean to set proper rule until the logger itself had the built-in ability to set such rule.

Once the logger gained that ability, I removed that temporary solution. Given that one context URL may match more than one tab, a proper solution would be to report to all the matching tabs. Whether this is a feasible solution in practice is not clear.

h1z1 commented 5 years ago

Not sure this is entirely Firefox unique as I run into something very similar with Chrome* browsers too. Simple example is you can't save a pdf from within pdfjs unless you allow Other even though you're already viewing it. I understand why it happens and know what "other" means in this context but it's not obvious. The save process appears successfull.

Example

Rules 
    https-strict: behind-the-scene false
    matrix-off: about-scheme true
    matrix-off: behind-the-scene true
    matrix-off: chrome-extension-scheme true
    matrix-off: chrome-scheme true
    matrix-off: moz-extension-scheme true
    matrix-off: opera-scheme true
    matrix-off: wyciwyg-scheme true
    noscript-spoof: * true
    referrer-spoof: * true
    referrer-spoof: behind-the-scene false
    * * * block
    * 1st-party cookie block
    * 1st-party css block
    * 1st-party frame block
    * 1st-party image block
    * 1st-party media block
    * 1st-party other block
    * 1st-party script block
    * 1st-party xhr block
    xiph.org people.xiph.org css allow
    xiph.org people.xiph.org image allow
    xiph.org people.xiph.org media allow
uBlock-user commented 5 years ago

you can't save a pdf from within pdfjs unless you allow Other even though you're already viewing it.

Because when you click the download button, a network request of type "Other" is fired by browser, so if other column is blocked, the download will fail. Viewing is not the same as downloading it.

The save process appears successfull.

It differs as per the browser. In Chrome, a save dialog will appear but the download will be marked Failed - blocked in chrome://downloads page.

In Firefox, the save dialog will fail with an error.

You can do one of two things either remove * 1st-party other block or add xiph.org people.xiph.org other allow

TT1 commented 4 years ago

I concur that this does indeed occur with Firefox 67.0 after upgrading to uMatrix 1.4.0 from 1.3.16. I discovered that allowing 'other' will allow successful downloading of items with a right-click. With 'other' NOT allowed, everything appears to work, but nothing is downloaded and it shows as a fail in the download history. This does not happen in 1.3.16. I'd prefer not to have 'other' allowed at all times, not knowing what might lurk there...

gorhill commented 4 years ago

Firefox tells uMatrix that the network requests are of type other, it's not uMatrix who decides to label the network requests as other.

Tiavor commented 4 years ago

I have this problem too, but I am trying to download pictures from reddit.com and it fails with all images that are hosted on reddit. I have everything, even "other", enabled for reddit.com. pressing the retry button in the download menu successfully downloads the image.

TT1 commented 4 years ago

Yes, that's exactly what I need to do, click the retry. It seems to only be a problem with media files, certainly ALL image files. The problem happened simultaneous with the last uMatrix update. I'm fairly certain the issue is with uMatrix because I made no other changes other that the update.

On 01/12/2020 01:38 AM, Tiavor wrote:

I have this problem too, but I am trying to download pictures from reddit.com and it fails with all images that are hosted on reddit. I have everything, even "other", enabled for reddit.com. pressing the retry button in the download menu successfully downloads the image.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/uBlockOrigin/uMatrix-issues/issues/92?email_source=notifications&email_token=AAKVUD3KDBHXCTCX4J6WQK3Q5LQH7A5CNFSM4GMGWK52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWVWOY#issuecomment-573397819, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKVUD6TRHQNQFZGQFLJ5D3Q5LQH7ANCNFSM4GMGWK5Q.

Tiavor commented 4 years ago

duh! reddit hosts the files on redd.it thus I need to whitelist that. found a few more sites where I have to whitelist them to be able to download them.

0xpr03 commented 4 years ago

I want to give another insight, as previous reports didn't focus so much on why this is annoying:

When you try to download the certificate from here listed as direct link you get the following error. (Right-click Save as..) image

uMatrix after this error (and also before) image

So even if you suspect umatrix to be the culprit, you won't see anything blocked and it's really irritating that a direct interaction will fail with such an non-explaining error.

Yes the browser is probably also to blame, but the current situation can lead you to believe something completely different is broken.

gorhill commented 4 years ago

you won't see anything blocked

You are supposed to look at the logger, without this I have no idea that what you report is either a Firefox or uMatrix issue. The logger will show you what is being blocked and in what context.

uBlock-user commented 4 years ago

you won't see anything blocked and it's really irritating that a direct interaction will fail with such an non-explaining error.

Tabless/bts requests do not appear in the popup UI. So open the logger and try again to troubleshoot.

0xpr03 commented 4 years ago

I'm sorry, yeah it's "other" again, so I guess firefox problems.. image

gorhill commented 4 years ago

It's a by-design browser behavior.

5t3f4n commented 4 years ago

I think I'm having the same problem but I'm not 100% sure. It is definitely related as it's the failed file saving, and I can see the blocked "other" request in BTS. However, if I click on retry download it instantly works – without changing any rules in uMatrix. I don't know what it does differently on 2nd try compared to 1st try. Maybe on 2nd try it's saving from cache instead of web? But seeing as the image is already loaded it should already save from cache the 1st try instead of re-downloading I guess.

recording

So I guess my question is this: why does it work on retry, and can I somehow automate the retry when it fails?

0xpr03 commented 4 years ago

@5t3f4n it's a browser API Design problem which can't be circumvented except by just disabling uMatrix for a short time. The download is seen as a different source and thus blocked.

5t3f4n commented 4 years ago

Yes I know why it's blocked in the first place. I'm just confused as to why it works on retry download without changing anything. Notice the uMatrix icon is still there – I didn't disable it or change any rules. A way to automatically click the retry-button would effectively solve my problem.

gorhill commented 4 years ago

I'm just confused as to why it works on retry download without changing anything

Maybe seeing what happens in the logger would provide an answer. The logger is really the tool to use when investigating what happens in uMatrix.

5t3f4n commented 4 years ago

@gorhill nothing exciting in the logger. 1st try: the "other" request is blocked just like all other cases ITT. Download fails. 2nd try: nothing appears anywhere in logger in any context. Download succeeds.

Screen Shot 2020-02-20 at 15 52 02

I don't understand why it works on retry since "other" is still blocked with uMatrix.

ThurahT commented 4 years ago

This behaviour was introduced for me in "Mozilla Firefox 68.5.0esr" version. Its a browser bug.

uBlock-user commented 4 years ago

Click the retry icon (right hand side of downloads list).

Retry button is a part of Firefox's UI, and any requests made by Firefox's UI are intentionally not sent through the webRequest API which blocking extensions like uBO or uMatrix listen to, this is also why you can no longer block Firefox's "check for updates" network request via any extension, nor any network requests internally made by Firefox. This was possible with Firefox Legacy, but not anymore after the Quantum update with v57.