search

OpenUserJS / OpenUserJS.org

The home of FOSS user scripts.
https://openuserjs.org/
GNU General Public License v3.0
858 stars 306 forks source link

"429 Too Many Requests" if you install two scripts that use the same library #1066

Closed cvzi closed 7 years ago

cvzi commented 7 years ago

If I try to install two different scripts that both use the same library, which is hosted on oujs, the second install will fail, because the library cannot be downloaded twice. The second time Greasemonkey tries to download the library file, oujs returns: "429 Too Many Requests".

I'm guessing this is a "feature" to prevent abuse, but maybe it's a little bit too strict?

Martii commented 7 years ago

I'm guessing this is a "feature" to prevent abuse, but maybe it's a little bit too strict?

It is to prevent abuse. Wait a few minutes. Since the next request would be "three" and then "four" and then so on and so forth this makes it Intended behavior. e.g. patience because otherwise it would be opening up a security hole.

Thanks for the report though.

Ref:

Martii commented 7 years ago

@cvzi

So I did a little digging and even if OUJS enables browser cache, which is what you need, Greasemonkey (GM) itself will override that at greasemonkey/greasemonkey/blob/696848a/modules/remoteScript.js#L606. This was first completed at greasemonkey/greasemonkey#1878 . Cc: @janekptacijarabaci

So basically nothing we do will override that from a search... which merits that brute force attack prevention. I'll keep digging but I seriously doubt we'll find a compromise. Sorry.

Cc: @derjanb Re: https://github.com/OpenUserJs/OpenUserJS.org/issues/957#issuecomment-236525483

This is with caching reenabled from a modified version of #894 and reverted in #895 . I've worked a solution around that regression issue but GM kills it dead with:

http://localhost:8080/install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
X-Powered-By: Express
Etag: "d2fc4dba73f26f6674a5af57690dfbafd4fadad9b8d385ce1b1af6e3d1af5b0b1fa66c390b29ee1f16a337d93145ee9d4f1dea3c0638f8029cb47760a78495d0"
Last-Modified: Sat, 01 Apr 2017 06:03:19 GMT
Cache-Control: public, max-age=86400
Content-Type: text/javascript; charset=UTF-8
Expires: Mon, 03 Apr 2017 02:58:31 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Sun, 02 Apr 2017 02:58:31 GMT
Connection: keep-alive
Transfer-Encoding: chunked

... notice the request Pragma and Cache-Control client side which prevents caching. We have to go with the most strict .user.js engine. GM itself could take a site down if reconfigured.

janekptacijarabaci commented 7 years ago

See also https://github.com/greasemonkey/greasemonkey/issues/2425

@Martii I understand why you are using it, but... AFAIK: There is not a simple solution.

I can not and do not want to speak for (Cc) @arantius , I (personally) will not create a PR (to turn cache on). This approach may have created more problems than good.

Martii commented 7 years ago

@janekptacijarabaci

No problem... just a courtesy notification of a site issue. Thought you might have some other idea from the .user.js engine side. I thought of how this would affect updating as well and the AOM updater would have to wait until the next cycle as well since that is probably rapid succession.

Anyhow thanks for the visit... perhaps a light bulb will turn on somewhere down the road.

janekptacijarabaci commented 7 years ago

I thank you for the information.

Martii commented 7 years ago

@derjanb Cc: @arantius

Random thought... So I asked about fingerprinting TM a while back and now with the Pragma and Cache-Control ... that's a fairly stable fingerprint for GM. Unfortunately that would give an unfair advantage to TM users if I put in a routine to do this detection plus that still doesn't cover all of the other .user.js engines. Yet another Catch-22.

arantius commented 7 years ago

If Greasemonkey continues to exist post-Firefox-57, it will be a brand new code base, and managing to continue to exist at all is my primary/only focus until that happens.

Martii commented 7 years ago

@arantius Understand... going to be a rough few months for everyone when Fx does its downgrade. Thanks for the read/response.

Martii commented 7 years ago

@derjanb I just tested your statement from https://github.com/OpenUserJs/OpenUserJS.org/issues/957#issuecomment-236525483 specifically:

The browser caching time depends on the headers sent along with the resource.

And it appears that current TM 4.2.7 under Chromium (Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36) mirrors what GM is currently doing with preventing browser cache. e.g. TM is also not allowing caching as shown through node's console.

Two successive calls with brute attack prevention disabled:

Modified since  undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 82.312 ms - -
Modified since  undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 74.190 ms - -

... behaves this way whether TM is enabled or disabled... unlike Fx (SM too) test with GM (with Port) disabled with same test pattern:

http://localhost:8080/install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://localhost:8080/scripts/Marti/RFC_2606%C2%A73_-_Hello,_World!
Connection: keep-alive

HTTP/1.1 200 OK
X-Powered-By: Express
Etag: "d2fc4dba73f26f6674a5af57690dfbafd4fadad9b8d385ce1b1af6e3d1af5b0b1fa66c390b29ee1f16a337d93145ee9d4f1dea3c0638f8029cb47760a78495d0"
Last-Modified: Sat, 01 Apr 2017 06:03:19 GMT
Cache-Control: public, max-age=86400
Content-Type: text/javascript; charset=UTF-8
Expires: Tue, 04 Apr 2017 01:04:17 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Mon, 03 Apr 2017 01:04:17 GMT
Connection: keep-alive
Transfer-Encoding: chunked

... and ...

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 83.202 ms - -
GET /images/favicon.ico 200 1.159 ms - -

... from nodes console... never changing on successive clicks of the Install button e.g. it pulled from the cache.

EDIT GM Port disabled and it pulls from browser cache even after an edit to the script source... that shouldn't be happening. Will re-verify the Headers sent.

Hmmm it would appear that the server is supposed to send no-cache directive ... received the expected results with GM Port disabled and successive Installs with node console of:

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 196.733 ms - -
did I even get here?
**************** NOT MODIFIED ********************
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 152.982 ms - -

... edited script here and reinstall...

did I even get here?
Modified since  Mon, 03 Apr 2017 01:23:44 GMT
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 159.452 ms - -

... will retry TM shortly...

did I even get here?
Modified since  undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 79.022 ms - -
did I even get here?
Modified since  undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 87.446 ms - -

... TM probably sending no-cache client side... retest without TM again...

did I even get here?
Modified since  undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 75.898 ms - -
did I even get here?
Modified since  undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 90.106 ms - -

Net results:

Ref(s):

derjanb commented 7 years ago

@Martii

I just tested your statement from #957 (comment) specifically:

The browser caching time depends on the headers sent along with the resource.

This statement was made in the conversation context of @require and @resource updates.

For regular script updates the no-cache header is used, but while thinking about this max-age=0, must-revalidate might be an option. I'll give it a try at the next beta version.

Martii commented 7 years ago

@derjanb

This statement was made in the conversation context of @require and @resource updates.

Confirmed however Chromium via TM doesn't seem to be using the cache. What's even more interesting is when I close The GM_config Unit Test the first, third time and successive page refreshes... TM attempts to redownload the .min.js lib. This is with the "Always" setting which you told me was:

Tampermonkey uses a default update interval of 7 days. The browser caching time depends on the headers sent along with the resource.

Our cache interval maxAge currently is 1 day but that will eventually go up.

GET /scripts/Marti/The_GM_config_Unit_Test 200 188.803 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 429.212 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 156.710 ms - -
pathname := /src/libs/Marti/GM_config.min.js
Cache-Control := undefined
Pragma := undefined
GET /src/libs/Marti/GM_config.min.js 200 489.123 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 155.141 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 156.295 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 154.723 ms - -
GET /src/libs/Marti/GM_config.min.js 429 4.649 ms - -

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36 with TamperMonkey 4.2.7

So your projects update mechanism seems to be still a bit buggy.

I'll be making a version of Greasemonkey Port that doesn't specify no-cache and see how that goes. Tests on OUJS dev went well without GM/Port enabled but will need to enable it (disabling the cache killer) and test it real time.

Martii commented 7 years ago

@arantius (sorry to bother you again if still listening... know you are busy elsewhere perhaps with webbymonkey) Cc: @janekptacijarabaci

These are the Live HTTP Headers with greasemonkey/greasemonkey/blob/696848a/modules/remoteScript.js#L606 commented out.

Initial install

https://openuserjs.org/install/sizzle/The_GM_config_Unit_Test.user.js

GET /install/sizzle/The_GM_config_Unit_Test.user.js HTTP/1.0
Host: openuserjs.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive

HTTP/1.1 200 OK
X-Powered-By: Express
Strict-Transport-Security: max-age=31536000000; includeSubDomains
Cache-Control: public, max-age=86400, no-cache, no-transform, must-revalidate
Last-Modified: Sun, 01 May 2016 22:47:56 GMT
Expires: Wed, 05 Apr 2017 01:56:33 GMT
Content-Type: text/javascript; charset=UTF-8
Etag: "8m9xuugi348448cckskkogc0okowwgk8ggcsgwsg48ww0kscoggscck080808kcko4cs0ksg0gw8c4sc804wokwwo0scw4848o8"
Date: Tue, 04 Apr 2017 01:56:36 GMT
Connection: close
----------------------------------------------------------
https://openuserjs.org/src/libs/sizzle/GM_config.min.js

GET /src/libs/sizzle/GM_config.min.js HTTP/1.0
Host: openuserjs.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive

HTTP/1.1 200 OK
X-Powered-By: Express
Strict-Transport-Security: max-age=31536000000; includeSubDomains
Cache-Control: public, max-age=86400, no-cache, no-transform, must-revalidate
Last-Modified: Mon, 27 Mar 2017 18:51:46 GMT
Expires: Wed, 05 Apr 2017 01:56:36 GMT
Content-Type: text/javascript; charset=UTF-8
Etag: "jaredo4fwe0wws8ocsswgs8s4cs40swg8k8wcsk0cg4ggoow88so4csgk00gwc4kc8w0s44c0g0wkc0owkksss0sc4g80sk0wk4"
Date: Tue, 04 Apr 2017 01:56:36 GMT
Connection: close
----------------------------------------------------------

Successive reinstall

https://openuserjs.org/install/sizzle/The_GM_config_Unit_Test.user.js

GET /install/sizzle/The_GM_config_Unit_Test.user.js HTTP/1.0
Host: openuserjs.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
If-Modified-Since: Sun, 01 May 2016 22:47:56 GMT
If-None-Match: "8m9xuugi348448cckskkogc0okowwgk8ggcsgwsg48ww0kscoggscck080808kcko4cs0ksg0gw8c4sc804wokwwo0scw4848o8"

HTTP/1.1 304 Not Modified
X-Powered-By: Express
Strict-Transport-Security: max-age=31536000000; includeSubDomains
Cache-Control: public, max-age=86400, no-cache, no-transform, must-revalidate
Last-Modified: Sun, 01 May 2016 22:47:56 GMT
Etag: W/"27-rVfB/Iv1jTj1VWtFK0s6+xv9+Hg"
Date: Tue, 04 Apr 2017 01:58:10 GMT
Connection: close
----------------------------------------------------------
https://openuserjs.org/src/libs/sizzle/GM_config.min.js

GET /src/libs/sizzle/GM_config.min.js HTTP/1.0
Host: openuserjs.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
If-Modified-Since: Mon, 27 Mar 2017 18:51:46 GMT
If-None-Match: "jaredo4fwe0wws8ocsswgs8s4cs40swg8k8wcsk0cg4ggoow88so4csgk00gwc4kc8w0s44c0g0wkc0owkksss0sc4g80sk0wk4"

HTTP/1.1 304 Not Modified
X-Powered-By: Express
Strict-Transport-Security: max-age=31536000000; includeSubDomains
Cache-Control: public, max-age=86400, no-cache, no-transform, must-revalidate
Last-Modified: Mon, 27 Mar 2017 18:51:46 GMT
Etag: W/"27-rVfB/Iv1jTj1VWtFK0s6+xv9+Hg"
Date: Tue, 04 Apr 2017 01:58:10 GMT
Connection: close
----------------------------------------------------------

Script updated and then reinstalled

(NOTE: Gm_config itself was not updated hence the 304 status)

https://openuserjs.org/install/sizzle/The_GM_config_Unit_Test.user.js

GET /install/sizzle/The_GM_config_Unit_Test.user.js HTTP/1.0
Host: openuserjs.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
If-Modified-Since: Sun, 01 May 2016 22:47:56 GMT
If-None-Match: "8m9xuugi348448cckskkogc0okowwgk8ggcsgwsg48ww0kscoggscck080808kcko4cs0ksg0gw8c4sc804wokwwo0scw4848o8"

HTTP/1.1 200 OK
X-Powered-By: Express
Strict-Transport-Security: max-age=31536000000; includeSubDomains
Cache-Control: public, max-age=86400, no-cache, no-transform, must-revalidate
Last-Modified: Tue, 04 Apr 2017 01:59:51 GMT
Expires: Wed, 05 Apr 2017 02:00:14 GMT
Content-Type: text/javascript; charset=UTF-8
Etag: "8wgmsisbez8cckw8ss4sw08g8o8goww4w8cskosckg0sskwcg4ks8gwssgkskogccko0wcggkwg0kckcswwoccgg4k80sggwggk"
Date: Tue, 04 Apr 2017 02:00:14 GMT
Connection: close
----------------------------------------------------------
https://openuserjs.org/src/libs/sizzle/GM_config.min.js

GET /src/libs/sizzle/GM_config.min.js HTTP/1.0
Host: openuserjs.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 SeaMonkey/2.40
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
If-Modified-Since: Mon, 27 Mar 2017 18:51:46 GMT
If-None-Match: "jaredo4fwe0wws8ocsswgs8s4cs40swg8k8wcsk0cg4ggoow88so4csgk00gwc4kc8w0s44c0g0wkc0owkksss0sc4g80sk0wk4"

HTTP/1.1 304 Not Modified
X-Powered-By: Express
Strict-Transport-Security: max-age=31536000000; includeSubDomains
Cache-Control: public, max-age=86400, no-cache, no-transform, must-revalidate
Last-Modified: Mon, 27 Mar 2017 18:51:46 GMT
Etag: W/"27-rVfB/Iv1jTj1VWtFK0s6+xv9+Hg"
Date: Tue, 04 Apr 2017 02:00:14 GMT
Connection: close
----------------------------------------------------------

greasemonkey-2017.04.03.beta-sm.xpi.zip

Cc: @cvzi ... give this a whirl for a test... may not be your browser (SeaMonkey) but may show you a thing or two. You will need to rename it and strip off the .zip (GH thing). Thanks.

janekptacijarabaci commented 7 years ago

Now: There is much incompatible PRs. Then I suggest (maybe):

https://github.com/greasemonkey/greasemonkey/blob/3.10/modules/remoteScript.js#L32:

var gIDFirefox = "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}";
var gIDPalemoon = "{8de7fcbb-c55c-4fbe-bfc5-fc555c87dbc4}";

https://github.com/greasemonkey/greasemonkey/blob/3.10/modules/remoteScript.js#L606:

  // But see also:
  // https://github.com/OpenUserJs/OpenUserJS.org/issues/1066
  if (((Services.appinfo.ID == gIDFirefox)
    && (GM_util.compareFirefoxVersion("42.0") < 0))
    || ((Services.appinfo.ID == gIDPalemoon)
    && (GM_util.compareFirefoxVersion("27.3.0") < 0))) {
    channel.loadFlags |= channel.LOAD_BYPASS_CACHE;
  }
Martii commented 7 years ago

@janekptacijarabaci

Re:

~ ... https://msdn.microsoft.com/en-us/library/aa365247.aspx#maxpath from short url alias of http://goo.gl/vDSk9

Perhaps http://www.ghacks.net/2016/05/27/microsoft-260-long-path-limit/ can shed some future light... maybe. ;)

Not sure what adding an extra UID would do even for SeaMonkey (SM). EDIT ... but I think I get your point for PM.

... suggest ...

I am pondering modifying GM Port to potentially collect a "white-list/black-list"... e.g. when the .user.js engine sends out a request as the client it should expect at the very least a no-cache from the server (that would be us at OUJS)... if present definitely don't send the LOAD_BYPASS_CACHE then... if there is ever not a no-cache response remove/add to the corresponding list. This doesn't really need a UI with SM/Fx so I think just a user preference... EDIT or perhaps in the config.xml.

derjanb commented 7 years ago

@Martii the latest beta is using must-revalidate for testing purposes.

Confirmed however Chromium via TM doesn't seem to be using the cache. What's even more interesting is when I close The GM_config Unit Test the first, third time and successive page refreshes... TM attempts to redownload the .min.js lib

Not sure what you are doing. I see resources and requires only being downloaded once.

Martii commented 7 years ago

@derjanb

EDIT

TM configuration for updating:

Start dev app with brute force attack prevention enabled...open script homepage...

$ node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /scripts/Marti/The_GM_config_Unit_Test 200 192.130 ms - -
GET /images/favicon.ico 200 6.013 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 485.122 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 158.952 ms - -

... then install button ...

GET /install/Marti/The_GM_config_Unit_Test.user.js 200 126.283 ms - -
GET /src/libs/Marti/GM_config.min.js 200 412.945 ms - -

... Refresh browser (1)...

GET /scripts/Marti/The_GM_config_Unit_Test 200 170.743 ms - -
GET /images/favicon.ico 200 2.961 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 161.879 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 163.818 ms - -
GET /src/libs/Marti/GM_config.min.js 429 11.571 ms - -

... Refresh browser (2)...

GET /scripts/Marti/The_GM_config_Unit_Test 200 158.102 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 160.729 ms - -
GET /images/favicon.ico 200 1.980 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 157.222 ms - -
GET /src/libs/Marti/GM_config.min.js 429 8.961 ms - -

... Refresh browser (3)...

GET /scripts/Marti/The_GM_config_Unit_Test 200 153.127 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 160.140 ms - -
GET /images/favicon.ico 200 2.083 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 154.571 ms - -
GET /src/libs/Marti/GM_config.min.js 200 303.468 ms - -

... Refresh browser (4)...

GET /scripts/Marti/The_GM_config_Unit_Test 200 155.633 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 153.865 ms - -
GET /images/favicon.ico 200 1.808 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 159.158 ms - -
GET /src/libs/Marti/GM_config.min.js 429 7.117 ms - -

... Refresh browser (5)...

GET /scripts/Marti/The_GM_config_Unit_Test 200 154.577 ms - -
GET /images/favicon.ico 200 2.421 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 160.579 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 152.613 ms - -
GET /src/libs/Marti/GM_config.min.js 429 7.834 ms - -

Issues: (1) Look at the GET request for /src/libs/Marti/GM_config.min.js... status code is next. Not utilizing any server side cache headers which is what I originally expected with the "Always" bit in TM... but you confused me, and still do, with:

Tampermonkey uses a default update interval of 7 days. The browser caching time depends on the headers sent along with the resource.

We now send headers... but with "Always" they are definitely ignored by TM... again I expect that with that terminology of "Always"... but overridden on server side caching.

(2) TM, and other engines, still send no-cache. This exact particular feature of TM is making it much worse for users on OUJS as they'll probably never be able to get at the libraries if they choose "Always" in the Settings especially since it's not acknowledging that OUJS has it cached now. As a result the top-level script will also never be able to be installed. The way the brute force dep works is if someone hits 429 a lot before the end of their timeout then the next time they'll get served and if more 429's happen then the length of the egg-timer is extended... repeat this multiple times and could be a very, very, very long time before a TM user can install a script (whether automatic or manual).

(3) Change in behavior between TM release and beta... 2nd browser refresh is acting better. 3rd browser refresh brute force prevention timed out so it got it after a few seconds of GM_config Unit Test being left open instead of happening 100% of the time after script execution completed earlier.

So basically the Beta seems to be better on consistency of what I consider "Always" to be... however no-cache still an issue with bypassing a Server that actually configures the cache properly. I am guessing the GM issue from 2014 was for USO (and possibly GH) and I know USO had all kinds of caching issues (e.g. scripts mismatches and meta mismatched)... probably just an issue with Ruby at the time.

So my proposal to TM is utilize the cache to start on libraries (what you call @externals) and if need be check to see if the target server sends no-cache back... if so then that's what is needed for a .user.js engine to be able to skip sending future no-caches to those servers thus utilizing server side cache. If the server ever stops sending no-cache back then TM would send a no-cache.

I'm working slowly on relaxing the brute force attack prevention when a .user.js engine does not send the no-cache directive.

Does this make more sense?

Martii commented 7 years ago

I think I see why MoonchildProductions/Pale-Moon#1002 was linked in here. Moz browsers and Chromium, when GM (TM indirectly possibly since Chromium supports the watered down version of .user.js natively) enabled, seems to have the same/similar issue... which is triggering an additional call to OUJS when brute is modified slightly I think... and no no-cache headers are present on 2nd call... and GM's URL interception fails. :\ What a mess... still stands on Intended Behavior for the time being.

EDIT @janekptacijarabaci PM is currently triggering three requests total once brute is in force (one user initiated and then two came from GMFork). GM/Port only triggers two. Technically speaking SM/Fx/PM with a .user.js engine enabled should plain stop, unless OUJS sends the try again header e.g. Retry-After, after the first 429 sent from OUJS.

Mozilla/5.0 (X11; Linux x86_64; rv:45.9) Gecko/20100101 Goanna/3.2 Firefox/45.9 PaleMoon/27.2.1 with 3.9.2Fork enabled nodejs log:

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 129.178 ms - -  # User initiated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 23.107 ms - -   # User initiated to invoke brute
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 6.878 ms - -    # Automatically initiated by .user.js engine
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.757 ms - -    # Automatically initiated by .user.js engine... End of Cycle

Mozilla/5.0 (X11; Linux x86_64; rv:45.9) Gecko/20100101 Goanna/3.2 Firefox/45.9 PaleMoon/27.2.1 with 3.9.2Fork disabled nodejs log:

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 139.721 ms - -  # User initiated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 12.423 ms - -   # User initiated to invoke brute... End of Cycle

Core ref:

janekptacijarabaci commented 7 years ago

You could try the following version:

Pale Moon 27.3.0a1 (unstable)

and

Greasemonkey for Pale Moon 3.12.1beta2 (ForkExperimental),

but with this change: blob/3.12.1beta2ForkExperimental/modules/remoteScript.js#L739

From: && (GM_util.compareVersion("27.3.0") < 0))

To: && (GM_util.compareVersion("27.3.0a1") < 0))

Martii commented 7 years ago

@janekptacijarabaci Thanks... fortunately my dev station is Linux so the Win binaries aren't a good suit here even in my Windows VM since I haven't quite figured out yet how to have those on the same IP network instead of their own private IP group. :) But I'l give the experimental a try in latest stable of PM.

P.S. Already checked the Linux site and it says:

Due to a recent server malfunction the Pale Moon for Linux website is largely unavailable. In the meantime, we have brought up the latest versions of the Browser and the Installer for download.

janekptacijarabaci commented 7 years ago

See https://linux.palemoon.org/download/unstable/

But actually it does not exist: Web server extended outage

And this one: http://web.archive.org/web/20170319043848/http://linux.palemoon.org/download/unstable/ is too old.

Martii commented 7 years ago

@janekptacijarabaci There's always you checking out dev here and running it :smile_cat: ... it's pretty easy... been meaning to make a gif or movie showing how it's done aside from the existing text on how to do it (Compiling and Creating a Local Environment) ... just need nodejs and ruby installed at bare minimum... maybe I'll go do that after a bit.

Martii commented 7 years ago

@janekptacijarabaci

Refigured out the 2nd call is in GM... GM gives up and passes it back to browser native to get again... this should ring a bell from #989 and greasemonkey/greasemonkey#2390. PM stable is trying twice with GMFork then passing it back to browser to reget again from what I can tell client side.

I think PR greasemonkey/greasemonkey#2415 is going to have to be rewritten to accommodate 429's right before the popup dialog for UI intercepts... not just background requests.

I had to put this diff in just to keep the OUJS UI from endlessly loading:

diff --git ./modules/util/showInstallDialog.js ./modules/util/showInstallDialog.js
index -..- -
--- ./modules/util/showInstallDialog.js
+++ ./modules/util/showInstallDialog.js
@ -53,10 +53,15 @@ function showInstallDialog(aUrlOrRemoteScript, aBrowser, aRequest) {
         browser.webNavigation.stop(Ci.nsIWebNavigation.STOP_ALL);
       } else if ((aStatus == 429) || (aStatus >= 500)) {
         // HTTP status code:
         // client errors (429 "Too Many Requests"), server errors
         aRequest.cancel(Components.results.NS_BINDING_FAILED);
+        var browser = aRequest
+            .QueryInterface(Ci.nsIHttpChannel)
+            .notificationCallbacks.getInterface(Ci.nsILoadContext)
+            .topFrameElement;
+        browser.webNavigation.stop(Ci.nsIWebNavigation.STOP_ALL);
       } else {
         aRequest.resume();
       }
     }
   });

... this diff (and subsequent SM xpi with LOAD_BYPASS_CACHE still enabled) is a little vague to the OUJS UI though. When the OUJS install button is clicked it basically does nothing hence why this should probably be done right before this stage:

popup dialog.

GM won't know the metadata yet but it could at least say a similar message to when a @require fails to load if missing like on a 404 like this:

gm failure

janekptacijarabaci commented 7 years ago

You could try the following version: Greasemonkey for Pale Moon 3.12.1beta3 (ForkExperimental),

but with this change: blob/3.12.1beta3ForkExperimental/modules/remoteScript.js#L752

From: && (GM_util.compareVersion("27.3.0") < 0))

To: && (GM_util.compareVersion("27.3.0a1") < 0))

Martii commented 7 years ago

@janekptacijarabaci I already know the cache reenabling works perfectly in SM with OUJS but that doesn't mean that some other site is configured the same way... so backwards compatibility is what I'm aiming for.

I tried your experimental (beta2) yesterday and that's why it needs the diff I mentioned above to stop the UI spinner in the browser... but it's very vague on what is happening. e.g. a user is going to tap the install button over and over and over and then open up an issue saying that they can't get the install button to work... and I'll say "intended behavior" and point them to a comment, issue, or document somewhere as to why. Rewriting the actual showing of the dialog to include the first, top-level, script would be a way to capture and show those error codes mentioned in PR greasemonkey/greasemonkey#2415 instead of being completely "silent" for just the first script. That code is wound up pretty tight but I think it's possible when it's expecting a non-silent action to make it include the first top-level script.

I also haven't checked out with git your branch to add the cache for PM yet... I will be doing that when PM's beta linux site is back up so I can test there as well. I think the third call to OUJS is what you fixed over there but won't know for sure until I have the Linux PM nightly in hand on this dev station.

I just wanted to say thank you all for coming back and we'll need to continue this after a weekend hiatus. If anything new is figured out please feel free to add here. :smile:

Martii commented 7 years ago

@janekptacijarabaci

Just tried beta 3 on my laptop... looking exactly what was needed for 429's in the UI... I'll have to do some testing when I get back to dev station to see if it's still hitting the server twice (especially with the relaxed brute commit not PR'd yet). Nice work!!!

Still oblivious to https://openuserjs.org/install/Marti/.user.js though... that's a 400. We may end up using some other 400 series codes as well.

Martii commented 7 years ago

Mirroring comment from this GMFork code point as when the branch goes away so do the comments:

@janekptacijarabaci

If a server sends something like 511 with full capabilities then GMFork will be cutting out the "SHOULD" aspect of the RFC 6585. Same goes for 401 with implied "must"s for example (Obviously there is some flexibility with 429 but we may eventually send the Retry-After and that really doesn't cut out the Server from a bounce back just the .user.js engine). e.g. I think it was better when you had it at https://github.com/janekptacijarabaci/greasemonkey/commit/5c92bd6bb756f79b1c42c3cdcfaf328697945bb1#diff-278d8aee3ff84e791438749205e97629R48

As much as I would like GM to handle all errors some just can't be accomplished with a generalization of >= 400 or >= 500... so probably should be exact status codes like you had initially. e.g. inclusive to "dead end" requests/responses.

janekptacijarabaci commented 7 years ago

Can you suggest which numbers of 5xx? (ad 4xx: https://github.com/OpenUserJs/OpenUserJS.org/pull/989#issuecomment-239525185)

Ad Retry-After - this could be useful :-)

Martii commented 7 years ago

Can you suggest which numbers of 5xx?

I'm going to have to go through all the well known ones, read up on their exact nature, and see for sure but for minimalist sake what you started out with of [429, 500] seems a good start for an inclusion list. GM Port is going to try out out [400, 429, 500] for now. 429 is special as I can already predict that GM itself is going to say that they don't want to maintain a retry routine... plus we might not send it (the Retry-After)... depends on how the gaming of the system proceeds. A bad aspect of most humans is usually when your down you usually can get kicked more often... which is something I'd rather not promote and part of the "politics" of Userscripting.

I would guess if it turns out there are more of >= 400's to be intercepted/notified than exceptions... GM Port (and anyone else willing to pipe in with test results on other sites) will probably utilize the inline function and use the list as an exception list instead of an inclusion list... time will tell. OUJS will still handle them as if no .user.js engine was present first then hope that the .user.js engine will do the necessary followup for human interaction sake.

Martii commented 7 years ago

@derjanb

TM BETA 4.3.5430

Simple one time install on OUJS dev, cancel, and immediate reclick of Install button now hits a total of three additional times (two with TM disabled) (Chromium still):

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 126.979 ms - -  # Initial click of install button then cancelled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 5.377 ms - -    # Second click of install button
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.272 ms - -    # Automatically hit
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.312 ms - -    # Automatically hit
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.927 ms - -    # Automatically hit
Martii commented 7 years ago

@janekptacijarabaci

Re: https://github.com/OpenUserJs/OpenUserJS.org/issues/1066#issuecomment-292540398

With code point janekptacijarabaci/greasemonkey/blob/3.12.1beta3ForkExperimental/modules/remoteScript.js#L751-L756 but applying this to SM... SM 2.39 uses the backend of Gecko 42.... so basically could use a modified test, or just bump the minVer to 2.40 and no test. SM has a tighter reign on minVersions than Fx and PM... so may do that once a little more testing happens with non-cache enabled.

Martii commented 7 years ago

@janekptacijarabaci Something doesn't quite seem right with beta4 but at least it's not as bad as upstream GM is going to be when I finish testing. An OUJS PR is complete and awaiting upload to detect the non-cacheable .user.js engines... GM Port rc3 (not uploaded yet) is behaving like I expect it to.

Look forward to a beta5 for PM? :)

Martii commented 7 years ago

@janekptacijarabaci

Never mind I see you aren't doing this until 27.3.0 and I'm on 27.2.1 until the Linux PM site is up fully. Sorry for the extra noise.

derjanb commented 7 years ago

@Martii

TM BETA 4.3.5430

Simple one time install on OUJS dev, cancel, and immediate reclick of Install button now hits a total of three additional times (two with TM disabled) (Chromium still):

Tampermonkey/Chrome sends Cache-Control:max-age=0, must-revalidate, If-Modified-Since:Sat, 01 Apr 2017 06:10:29 GMT and If-None-Match:"pVfmR7xIQKu0Cs8aagOeoiCyEiM8qY0aiwKuYQyAKMuA2y2O42isaYqeKsaCGW4o8cQa2AI0OW2kOmCKeSiSwi"

So in my opinion the problem is OUJS responding with 429 instead of 304 or am I missing something?

image

Martii commented 7 years ago

@derjanb I'll have to retest dev backed up a bit since the time that I mentioned that there was 2 deploys. When the full cache check is released a few hours ago on dev TM Beta performed as it should... but have to retest again before deployment again on that PR.

Martii commented 7 years ago

Yikes... even worse today with current HEAD

GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 86.871 ms - -   # Install button then cancel
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.399 ms - -    # Install button... expected 429
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.092 ms - -    # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.685 ms - -    # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.096 ms - -    # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.667 ms - -    # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.296 ms - -    # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 112.447 ms - -  # Automatic from Chromium Browser with TM enabled... but a few seconds later

... 429 instead of 304 or am I missing something?

Once a 429 is hit we do nothing plus we don't force a browser to regenerate a request (that is the Retry-After). This might be a Chromium glitch too or a difference between dev and pro since on dev we do send the Retry-After.

P.S. @derjanb Make sure your cache is clear as if a new user... in a new browser session.

Martii commented 7 years ago

@derjanb Here's a video of it. Be sure to watch to the end... all I was doing was other things on my other monitor and occasionally hovering back over Chromium... and then the final 200 came out of nowhere. These are all GETs as you can see and no PUTs from us.

Chromium with TM.zip (H.265 mp4 contained within) (~1.2MiB) active on d843552d83fa72a9b915dd04f114b12abae4190d

Martii commented 7 years ago

@cvzi

So if you are using GM... I would suggest trying GMFork for now (will have an issue on the release channel with code signing though for Fx); GM Port is updated in rc3 for SM; TM Beta should be okay; VM never had the issue of cache disabling; QupZilla tests good.

Ref:

Martii commented 7 years ago

@derjanb So with a5bee24825c32af473e318b07d3dce10384e7722 and TM Beta 4.3.5430 still getting this:

$ FORCE_SCRIPT_NOCACHE='true' && node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 144.207 ms - - # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 79.735 ms - -  # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.809 ms - -   # User initiated install to invoke protection... expected
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.926 ms - -   # Unexpected automatic response from Chromium via TM Beta (1)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.932 ms - -   # Unexpected automatic response from Chromium via TM Beta (2)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.915 ms - -   # Unexpected automatic response from Chromium via TM Beta (3)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.824 ms - -   # Unexpected automatic response from Chromium via TM Beta (4)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 2.274 ms - -   # Unexpected automatic response from Chromium via TM Beta (5)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 81.032 ms - -  # Unexpected automatic response from Chromium via TM Beta after a wait period (6)

... and ...

$ FORCE_SCRIPT_NOCACHE='true' && node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 131.704 ms - - # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 90.924 ms - -  # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.627 ms - -   # User initiated install to invoke protection
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.889 ms - -   # Unexpected automatic response from Chromium with no .user.js engine enabled (1)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.687 ms - -   # Unexpected automatic response from Chromium with no .user.js engine enabled (2)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.809 ms - -   # Unexpected automatic response from Chromium with no .user.js engine enabled (3)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.631 ms - -   # Unexpected automatic response from Chromium with no .user.js engine enabled (4)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 96.271 ms - -  # Unexpected automatic response from Chromium with no .user.js engine enabled after a wait period (5)

... I would have to say that Chromium has a serious issue with not paying attention to response headers especially when it's told "no" from a server. Might keep that in mind to see if TM can work around the browser sending out so many... and one extra for TM when enabled... this is all when we protect the server and it's been verified in many other browsers as not our issue. I understand it might be out of TM's hands, other than the single extra GET call, but when OUJS is in lockdown TM users won't be able to get much done.

As I said earlier if TM doesn't send no-cache and it's cacheable and we don't protect (since it pulls from client memory or disk cache) then TM Beta is peachy. *shrugs*

$ FORCE_SCRIPT_NOCACHE='false' && node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 117.496 ms - - # User initiated install... not cached yet... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 72.793 ms - -  # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 75.057 ms - -  # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 71.921 ms - -  # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 72.793 ms - -  # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 75.057 ms - -  # User initiated install... cached pull... from Chromium via TM Beta
Martii commented 7 years ago

Btw what is in the url (address bar) of: http://localhost:8080/install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js#bypass=true? :question: ... we never add #bypass=true... a web search didn't yield much other than this.

If I was to take a guess it's possibly the last 200 after a few seconds after TM and the extension framework gives up... but just a guess.