"Malware domains" list fails to update

[andreygursky]

Prerequisites

• [x] I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
  • [x] I also searched the existing issues at https://github.com/gorhill/uMatrix/issues
• [x] This is not a support issue or a question
  • Support issues and questions are handled at /r/uMatrix
• I tried to reproduce the issue when...
  • [ ] uMatrix is the only extension
  • [x] uMatrix with default lists/settings
  • [ ] using a new, unmodified browser profile
• [x] I am running the latest version of uMatrix
• [x] I checked the documentation to understand that the issue I report is not a normal behavior
• [ ] I used the logger to rule out that the issue is caused by my ruleset

Description

Since a couple of months uMatrix is not able to update the default "Malware domains" [1] list on a 64 kbit/s connection. The reason is a timeout. I'm not sure whether the size of that file has grown rapidly or the server stopped to deliver this text file in a compressed format.

The default 30 seconds timeout is hardcoded here [2]

// src/js/assets.js
        xhr.timeout = 30000;

It looks like it was previously adjustable [3]:

var xhrTimeout = vAPI.localStorage.getItem('xhrTimeout') || 30000;

and this functionality has been restored recently [4]:

// src/js/assets.js
    const timeoutAfter = µMatrix.rawSettings.assetFetchTimeout * 1000 || 30000;

Thus I'd like kindly to ask to make at least this only change available (or if not possible, then raise the hardcoded default timeout temporary) in the next release.

Thanks.

A specific URL where the issue occurs

Steps to Reproduce

1. Ensure your internet provider throttles the connection bandwidth to 64 kbit/s.
2. Ensure the mentioned "Malware domains" [1] list is outdated.
3. Try to update assets.
4. The update for the mentioned list will always fail.

Ruleset

Supporting evidence

$ wget https://mirror.cedia.org.ec/malwaredomains/justdomains
$ ls -l justdomains 
-rw-r--r-- 1 594673 Mar  4 23:42 malwaredomains/justdomains

More than 74 seconds are needed to download this file at 64 kbit/s.

Your environment

• uMatrix version: 1.4.0
• Browser Name and version: Firefox 74.0
• Operating System and version: Debian GNU/Linux testing (Bullseye)

[1] https://mirror.cedia.org.ec/malwaredomains/justdomains [2] https://github.com/gorhill/uMatrix/blob/1.4.0/src/js/assets.js#L116 [3] https://github.com/gorhill/uMatrix/commit/c2d7096500c11b72888b4d901b64185f9cc7b9d1 [4] https://github.com/gorhill/uMatrix/commit/9b292304d33a44465922200efa5f8b378d0f9604

[uBlock-user]

@andreygursky Open 'More' pane and set assetFetchTimeout to 60 or some bigger value, that should help.

[gorhill]

Please try the dev build of uMatrix, it's better equipped to deal with slow connection as it takes into account progress events when downloading assets in order to determinate whether there is a time out condition.

[andreygursky]

@uBlock-user, thanks, I've tried this but my changes were ignored. Then I've figured out the option is too new for uMatrix 1.4.0.

@gorhill, updated. Now it is better, thanks!

as it takes into account progress events when downloading assets in order to determinate whether there is a time out condition.

Indeed, the issue is actually not that no data is transmitted for 30 seconds but only rather slow. From what iftop shows, once a first timeout hits, Firefox seems to continue downloading, but since uMatrix 1.4.0 receives the timeout event, it starts the same download from a mirror, then I see 2 concurrent downloads in iftop, which makes it even worse.