Closed Yonezpt closed 2 months ago
Can you update the extension to be more reliable in this aspect?
Unfortunately not. The (Tampermonkey) content script running at the page needs to get the scripts that should executed from the background page. This is done with some asynchronous communication that doesn't take long, but some pages are faster.
The solution for this is Chromium issue 63979 [1]. Once this is fixed the background page can actively inject the runtime env + all scripts into the matching frames before the page script runs.
[1] https://code.google.com/p/chromium/issues/detail?id=63979
So it is a browser limitation, not extension limitation, good to know.
For the moment I managed to make the script runtime more reliable from my userscript (or at least less dependent on Tampermonkey's timing), was working on trying to find a better solution since I posted here, so for now it seems like I will be able to continue extending support in my userscript for Tampermonkey. I hope they fix that issue because this will allow me to reduce code size and overhead.
Thanks for the quick response.
@derjanb
The solution for this is Chromium issue 63979 [1]. Once this is fixed the background page can actively inject the runtime env + all scripts into the matching frames before the page script runs.
looks like the issue was fixed
Unfortunately there is a new showstopper... https://bugs.chromium.org/p/chromium/issues/detail?id=598896
Currently Tampermonkey adds a lot of overhead in its content script as seen in DevTools Timeline profiler, and because of that document-start userscripts are postponed until the initial spike of webpage parsing is complete, which can easily take 1 second or more.
chrome.runtime.sendMessage
was used directly in top level code.I don't know the architecture of TM but when I see fancy beautiful OOP wrapper, I know immediately with 100% certainty that this code can be "flattened" and become faster. Sometimes for just a few milliseconds, sometimes several/many times faster.
For example, Stylish-chrome also injects user-defined styles at document-start, and does it reliably in time, before the page is shown, so no FOUC occurs. It's because 1) Stylish immediately sends the message to its background page, 2) the styles are cached in memory to avoid DB access overhead, 3) so the response is sent back immediately, not in a new message that may be postponed by browser.
Currently Tampermonkey adds a lot of overhead
Styles don't need to be separated at the page context and they need no runtime environment with powerful GM_* functions to work, they can simply be injected into the page.
For example, Stylish-chrome also injects user-defined styles at document-start, and does it reliably in time, before the page is shown, so no FOUC occurs. It's because 1) Stylish immediately sends the message to its background page, 2) the styles are cached in memory to avoid DB access overhead, 3) so the response is sent back immediately, not in a new message that may be postponed by browser.
TM also caches the scripts and sends them back immediately. However, since extensions run in their own process and the communication is asynchronous there is no guarantee that the page didn't finished loading. This means Stylish also has that race condition. If I'm wrong, then please tell me how synchronous message passing works. :)
how synchronous message passing works.
Of course, it's not synchronous. I meant Stylish gets the data sooner, the response returns in 20-30ms:
But anyway, I was sloppy in interpreting the profiler graph. TM also sends the request in the first milliseconds (I was wrong about that), but it doesn't help much, because TM's background page takes 150ms to send the response so browser has time to finish parsing DOM:
So even though I was mistaken in diagnosing the problem, maybe you can make the background page respond faster (100ms is hell of a lot of time) as it seems crucial to the browser.
Maybe you can prepare the list of userscripts to be injected for a given URL in webNavigation.onBeforeNavigate
listener and keep this list ready to be sent when content script asks for it. Of course, since onBeforeNavigate may be canceled, the list should autodestroy in like 1 second.
Also, you can try repeatedly sending (using setInterval) the list of userscripts in webNavigation.onCommitted
- it'll fail the first time since no content script is running at this point but one of the subsequent attempts should succeed, supposedly, and the content script will receive the data much sooner.
Maybe you can prepare the list of userscripts to be injected for a given URL in webNavigation.onBeforeNavigate
TM uses a blocking webRequest callback to do this.
Also, you can try repeatedly sending (using setInterval) the list of userscripts in webNavigation.onCommitted
I just tried this and it's working good so far. I don't know how large Stylish's user styles usually are, but user scripts and their resources might get really big. This solution would cause a lot of overhead, because all data is sent (and stringified) at least two times or more. So I'm not sure if this is desired behavior. Maybe I'll put this behind a config option.
Some userstyles are really really big, but I don't have any of those.
What about preventive sending only of document-start script by default? From what I saw, those are relatively rare, but their functionality really depends on being injected ASAP. Or maybe some hardcoded(?) limit like 10-100kB.
The latest beta version got an option to enable the fast injection mode. (You have to enable the advanced config mode to see it.) I'd appreciate if you could do some testing and also observe whether there are differences regarding the CPU utilization.
Yay!!!! Now it completes before the very first page paint.
I don't see any CPU difference (using Process Hacker) with 6 document-start scripts (200kB) and 2 document-idle on this page out of ~50 installed overall. Peaks are at 4-5% which on i7 means 30-40% of one hyperthreaded core.
It takes almost 200 ms though, which is not noticeable for a webpage (moreover the images/css/js are still being fetched), but still seems big for my 6 userscripts. The same duration as always but previously it occurred after the first paint so I was seeing FOUC, then indeterminate pause, then 200ms pause for TM's message, then userscripts did their job.
Are you fast-injecting just the document-start scripts or all the applicable scripts for the page?
I'm looking at userscripts with large amount of info stored via GM_setValue, and I wonder if it's worth using a threshold.
Fast injection for @document-start:
Tooltip: This might cause higher CPU usage. Tiny script is less than 10kB of code and data combined, small: 100kB.
Sorry for the late reply.
Are you fast-injecting just the document-start scripts or all the applicable scripts for the page?
For the moment I prefer to have as less code paths as possible here. Therefore all scripts are injected the same way. My secret hope is that the stringified object is cached and re-used at the second message. I'm curious to see what happens once TM BETA has the fast mode enabled by default.
My secret hope is that the stringified object is cached
I don't see caching in the chromium source code. On the other hand, serializing/parsing is supposedly super fast (less than 10% of time spent in TM according to timeline profiling).
For the moment I prefer to have as less code paths as possible here
The thing is, some (many) sites load/create/modify resources needed to display a page in js code, which gets delayed by the time spent in TM. And that time depends on the message size and amount of script contexts created (my test on i7 CPU shows 40ms for one small script, 100-200ms for 10 medium-sized scripts). In other words, TM delays the first page paint on dynamically loaded sites especially in case of subsequent visits when most of the page was cached by browser. On a slower computer the delay is bigger, I guess.
Related: chromium feature request, Firefox feature request. An interesting hack currently possible to implement: abusing document cookies. @derjanb, what do you think?
@tophf Interesting approach. I'll check that. Especially if there are length limitations. Thanks for the hint.
@tophf It's pathetic. This hack is working in general, but there seems to be a cookie size limit. Small cookies are accessible at document-start, but larger ones are simple ignored. 😭 http://browsercookielimits.squawky.net/
@derjanb maybe I'm wrong, but what if TM inject CSP header and block all possible scripts? At the very beginning?
I don't know how TM (and extensions) work here, but at the moment I want to block everything on the page and replace site functionality with my script.
Maybe it is possible to implement this as header option? Like:
// @csp-header script-src: 'blob:' 'sha256-qznLcsROx4GACP2dm0UCKCzCG+HiZ1guq6ZZDob/Tng='
// @csp-header script-src: https://cdn.com
So, everything except blob:
URIs, scripts from cdn.com
and something matching that hash will be blocked.
This is not right solution for original problem, but this at least add some flexibility for cases when you want to override page scripts completely.
In Greasemonkey I can just do on document-start
something like this:
Object.defineProperty(window, 'importantPartOfTheirScripts', {
enumerable: false,
configurable: false,
writable: false,
value: undefined
});
But in Tampermonkey this does not work...
It's working now. Please check the "Instant" injection mode at the latest beta version 🤓 (config mode needs to be set to "Advanced" in order to see this setting.)
Wow, blob URL + sync XHR, that's ingenious! It shows a warning in the console about sync XHR though.
Ideally, document_start scripts should be injected separately to reduce the time needed to create script sandboxes, but I obviously don't have TM user statistics (Chrome is awesome in that regard) so maybe it's just an edge case. On my fast desktop PC (i7 4GHz) it takes 70ms for 10 scripts active on github.com as measured in content.js of a locally modified TM:
var e = function() {
console.time(1);
var r = c.dispatchEvent.apply(c, arguments);
console.timeEnd(1);
return r
}
The injected stringified object (arguments[0].attrName) is 500kB.
Captain Obvious here. I searched "instant" here to mention the warning in the console, but I expected you were well aware. There was a noticeable difference with a few scripts using "fast" inject mode, so this new method that's even faster is great news.
I'm working on a script which has to be executed before page scripts. Is it reliable to use Inject Mode = Instant
? Should I suggest the user turn it on?
If it is not reliable, I would make an extension instead. I'm not even sure whether content scripts are fast enough.
Sadly instant injection doesn't work for iframes, the dom gets loaded before the scripts run This was incorrect, it wasn't working because I was loading a local file. When it was saved to TM directly it worked fine.
@derjanb, Chrome 67 (66 too, probably) has broken instant mode, see https://crbug.com/825111. Hopefully they'll fix it before the release, but maybe not, so it might be a good idea to implement a fallback to normal mode and print a message in the console or show it in dashboard.
@tophf Thanks for the hint.
Apparently there is a way to run page code even earlier than with currently available "Instant" mode. Here is an example page: view-source:https://sinoptik.com.ru/10-%D0%B4%D0%BD%D0%B5%D0%B9 And in particular:
In the page header. I have a script which changes a few functions in XMLHttpRequest API to check URLs passed into it and depending on URL ignores all following interactions with that object, yet they still manage to run their code before mine and in the result my wrapper blocks nothing.Maybe they create a dummy iframe with srcdoc + document.write (see https://crbug.com/760954) and use its XHR.
They likely did. At least I already encountered similar script with that trick. However, I wrapped HTMLIFrameElement.prototype.contentWindow getter as well and it wraps APIs within an IFRAME objects as soon as someone attempts to access contentWindow in it.
Here is my script: https://greasyfork.org/en/scripts/19993-ru-adlist-js-fixes/code?version=608539 In particular rows 1382 ~ 1472.
Yes, they do access contentWindow and use XHR from that window. However, they didn't use that srcdoc trick. I've attached partially deobfuscated code which they are using. Starting from row 540 they either access existing frame on the page or going to contentWindow of IFRAME which they created right above on rows 527~539. Then they call function qo() (row 402) with that contentWindow as a parameter and create object with consturctor c(iframe.contentWindow) at row 429. Constructor c() defined on row 58. It stores link to original XMLHttpRequest and methods "open", "send" and "getAllResponseHeaders" from prototype of that object. Then it defines single method "r" which creates new XHR request and sends it using stored link.
As I udnerstand my code has to replace XHR methods within that contentWindow as soon as they accessed it.
Any updates on this? As I told it doesn't seems like they are using srcdoc
trick yet their code definitely runs before code injected by Tampermonkey. Is there a way to guarantee that injected code runs first? Since issue with sinoptik.com.ru
/ sinoptik.ua
still remains. I can easily catch their first and any following requests on sites like strana.ua
or korrespondent.net
where they are using normal script without data-url (<script type="text/javascript">!function(){...}();/*13a60edeb2527b43875c9f485a6bb7339b3b0efe*/</script>
), but I can do nothing on sinoptik. Most likely due to this particular trick with data-url.
@lainverse, as a workaround, try rewriting the page html completely (example).
That breaks site entirely since Chrome's blocks all third-party scripts injected with document.write.
Stuff like this:
VM4770:2 A parser-blocking, cross site (i.e. different eTLD+1) script, https://sinst.fwdcdn.com/js/1/jquery-1.10.0.min.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.
Currently circumvented by throwing an error when script which they load attempts to access specific window property and that stops its execution, but they can easily circumvent this as well.
Hi @tophf ,
Could you please consider enablid "Instant" injection mode by default? It works surprisingly well both in Chrome and Firefox as I can see and my script always run before anything on the page. Is there any reason to keep using old injection method as default one?
Apparently that code from sinoptik.ua
were using a trick I was not aware about. If you make an IFRAME, attach it anywhere in the document and assign it a name
it's window object will immediately become available as a property in a global context. The important part is that to access that window IFRAME's contentWindow getter (at least the wrapped one) is not used.
Example of their trick:
let str = '_'+Date.now();
let div = document.createElement('div');
div.innerHTML = '<iframe name="'+str+'"></iframe>';
let frm = div.children[0];
document.head.appendChild(frm);
if (window[str])
store_pointers_to_clean_API(window[str]);
else
[here they were trying to access IFRAME's contentWindow if method above failed]
document.head.removeChild(frm);
Last month I implemented a wrapper both for innerHTML
and appendChild
to check for IFRAME objects attached to a page in the result set to call for my API wrapper before page would do so and had no issues with it since then.
Is there any reason to keep using old injection method as default one?
There are no known issues at the moment, but the "Instant" way of forwarding information to the page is very very hacky.
In what way?
Le ven. 2 nov. 2018 à 21:16, Jan Biniok notifications@github.com a écrit :
Is there any reason to keep using old injection method as default one?
There are no known issues at the moment, but the "Instant" way of forwarding information to the page is very very hacky.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Tampermonkey/tampermonkey/issues/211#issuecomment-435485722, or mute the thread https://github.com/notifications/unsubscribe-auth/AF8AO_otQWM7ywEwIuHPvGlmO5uSGv7tks5urKgkgaJpZM4EDmD- .
In what way?
Well, it works and works quite reliably even if it's hacky. Does Chromium team aware that removal of this feature will affect the only currently available way to run user scripts at document_start?
https://mathiasbynens.be/demo/opener bypasses Instant mode, Its a part of https://mathiasbynens.github.io/rel-noopener/ test.
// ==UserScript==
// @name noopener
// @include http://*
// @include https://*
// @run-at document-start
// ==/UserScript==
unsafeWindow.window.opener = undefined;
fails to inject in time. "Deprecation] Synchronous XMLHttpRequest on the main thread is ..." warning missing in console.
Whats interesting target="_blank" rel="noopener" and rel="noreferrer" links also bypass Instant mode, but only when loaded first time?!?
// ==UserScript==
// @name noopener
// @include http://*
// @include https://*
// @run-at document-start
// ==/UserScript==
unsafeWindow.window.opener = "123";
fails to inject in time for "Click me!!1 (now with rel=noopener)" and "Click me!!1 (now with rel=noreferrer-based workaround)" links on https://mathiasbynens.github.io/rel-noopener/ BUT if you reload already opened subpage Instant mode triggers correctly!
Edit: removed // @grant none
from example script, was disabling unsafeWindow
@raszpl it works for me with "instant" injection mode It's at the bottom of the settings page when "Advanced" mode is enabled (at the top). Unfortunately it isn't default mode, so you have to set it manually.
Hmmm, I only tested on Vivaldi 2.9.1735.3 (Official Build) (32-bit). Trying Opera 64 ... and it works reliably. It is a Vivaldi bug then /sad face/, slim chance of getting Vivaldi to fix that :( I guess Ill have to convert one of my scripts into dedicated extension.
@raszpl Works fine for me with Vivaldi 2.10.1745.23 (Stable channel) (32-bit). Worked as fine with 2.9 before I updated it. Try to reinsall Tampermonkey. I've encountered similar problems with it which got resolved by reinstall.
I dont mean it doesnt work at all, just in listed cases. Its highly improbable you tested it in this particular way before upgrading. Nevertheless I went back to Vivaldi 2.8.1664.40 (Stable channel) (32-bit) on another computer and
rel=noopener
and rel=noreferrer
Instant mode does not trigger when tab is opened, only on consecutive tab reload.The way to test it is: 1 install
// ==UserScript==
// @name noopener
// @include http://*
// @include https://*
// @run-at document-start
// ==/UserScript==
unsafeWindow.window.opener = "123";
2 go to https://mathiasbynens.github.io/rel-noopener/
3 click on 3rd and 4th Click me!!1 links
4 Opened tabs display different text depending on the value of !(window.opener). Look at the text, reload tabs and observe text changing - indicating window.opener = "123"
wasnt injected early enough the first time.
I will test Vivaldi 2.10.1745.23 now ... and can confirm 2.10.1745.23 under Win7 acts like 2.8 (bad injection on noopener/noreferrer). Vivaldi definitely is screwing with something internally, not the first time, weak chances of them getting around to fixing it. Nothing derjanb should worry about, just another small browser bug :/
Edit: funny, https://mathiasbynens.be/demo/opener still doesnt work under 2.10 on older, upgraded installation (couple extensions, history going back 2 years). Ill play with deleting cache/extensions to isolate the culprit. Edit2: that was quick. My \User Data\Default\Service Worker\ directory was 650MB in the 2 year old install. Deleted whole thing and https://mathiasbynens.be/demo/opener finally gets the "[Deprecation] Synchronous XMLHttpRequest on the main thread is ..." warning in console. So it appears my issue was related to https://github.com/Tampermonkey/tampermonkey/issues/756. No clue about the mechanism that would do this, some 'clever' Vivaldi tab hibernation caching maybe?
Ah, I see. Indeed, it doesn't work in 3rd and 4th cases. Probably because page is loaded so fast that extension doesn't even recieve necessary events in time as in that bug you linked. I've re-instealled Vivaldi from scratch and remove entire User Data folder and still no message in console. Guess it's an offtopic at this point.
@derjanb it's actually possible to make the instant injection work on superfast pages: if the TM's cookie is missing when the content script runs, it should make a sync-xhr to which the background script will answer with a string containing a blob URL, then the content script will do the usual sync-xhr on the blob URL. For some reason the blob can't be fetched directly so both xhr calls are needed.
The only bad thing about this approach is that it's not configurable using storage so TM will have to use chrome.declarativeContent
API with RegisterContentScript
action to add a mini content script with this check.
Found another case where Instant doesnt work, this time both Vivaladi and Opera, CSP seems to be the culprit. https://www.troyhunt.com/promiscuous-cookies-and-their-impending-death-via-the-samesite-policy/
// ==UserScript==
// @name 123
// @match *://*/*
// ==/UserScript==
alert("You wont see me.");
results in
VM338:65 Syntax error @ "123"!
##########################
JSHINT output:
##########################
EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' c.disquscdn.com disqus.com troyhunt.disqus.com www.google.com www.google-analytics.com www.gstatic.com cdnjs.cloudflare.com platform.twitter.com cdn.syndication.twimg.com syndication.twitter.com gist.github.com/troyhunt/ 'sha256-dblwN9MUF0KZKfqYU7U9hiLjNSW2nX1koQRMVTelpsA=' 'sha256-4JqPqO/eQLWuWw1AE7dCvI9hPwiBcw0gy7uoLqS0ncg=' 'sha256-q7PyCIWqx04xiOpJNrqiwsSEIdeaqyhUMFifRsUwUDk=' cdn.report-uri.com".
at new Function (<anonymous>)
at Object.E_c (<anonymous>:3:393)
at la (eval at exec_fn (:1:107), <anonymous>:64:272)
at Object.create (eval at exec_fn (:1:107), <anonymous>:76:116)
at e (eval at exec_fn (:1:107), <anonymous>:16:355)
eval @ VM338:65
eval @ VM338:26
F @ VM336:9
V @ VM336:10
e @ content.js:6
(anonymous) @ content.js:7
(index):1 GET https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/fonts/fontawesome-webfont.ttf?v=4.7.0 net::ERR_BLOCKED_BY_CLIENT
VM338:65 Uncaught EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' c.disquscdn.com disqus.com troyhunt.disqus.com www.google.com www.google-analytics.com www.gstatic.com cdnjs.cloudflare.com platform.twitter.com cdn.syndication.twimg.com syndication.twitter.com gist.github.com/troyhunt/ 'sha256-dblwN9MUF0KZKfqYU7U9hiLjNSW2nX1koQRMVTelpsA=' 'sha256-4JqPqO/eQLWuWw1AE7dCvI9hPwiBcw0gy7uoLqS0ncg=' 'sha256-q7PyCIWqx04xiOpJNrqiwsSEIdeaqyhUMFifRsUwUDk=' cdn.report-uri.com".
at new Function (<anonymous>)
at Object.E_c (<anonymous>:3:393)
at la (eval at exec_fn ((index):1), <anonymous>:64:272)
at Object.create (eval at exec_fn ((index):1), <anonymous>:76:116)
at e (eval at exec_fn ((index):1), <anonymous>:16:355)
It only craps out with Instant and default // @run-at document-end
. // @run-at document-start
works fine.
Ah, I see. Indeed, it doesn't work in 3rd and 4th cases. Probably because page is loaded so fast that extension doesn't even recieve necessary events in time as in that bug you linked
I retested in (64-bit) Vivaldi and it works reliably, Ill try Opera 32bit ... no I wont, they dont let you directly download 32bit portable version :(. Now I wonder if its really Vivaldi specific, or maybe chrome 32bit build bug.
Well, that didnt last long :(. Inject Mode: instant
stopped working in both 32 and 64bit builds of Vivaldi. I deleted \Service Worker\ directory just to be sure, didnt fix it this time. Tested:
missing blob:chrome-extension://gcalenpjmijncebpfijmoaglllgpjagf/random-string
XHR in Network tab
Does it still work in newest Chrome builds? Edit: replying to myself:
instant works fine in Chrome 80.0.3987.149 (Official Build) (64-bit)
instant doesnt work in Chrome 81.0.4044.83 (Official Build) beta (64-bit)
so after all its not just Vivaldi :(
EDIT: This was #1083 fault, "Chrome ignores extensions on first page load", run-at document-start works fine after Extensions get a chance to register with the browser.
This can be used by web page to mess with violentmonkey internals using window.open()
or <iframe>
tags.
example: window.open("/").Array.prototype.forEach=console.log;
Currently when using
document-start
with Greasemonkey my script runs when the HTML element is created, before the Head element is loaded with new elements. However, the same isn't happening with Tampermonkey, resulting in an unreliable behavior which I was compelled to try and fix by forcing the page to reload in an attempt to get the script running when it should.With this simple example script for YouTube I get mixed results:
If I open a YouTube page for the first time it usually runs the script right at start when I need it, but if I open a video in a new tab (midle-click on link, for example) then sometimes this isn't true. On some cases the entire head is fully loaded and the body already started loading before the script could run, which can be confirmed by inspecting the page code after the script makes it stop. Also I witness often that the head already contains scripts that I need to load after my script runs, not before, which can be seen in the console.log.
The worst case is when I open a new window via
window.open()
which the script fails to run when it should every single time for up to 10 or 20 times in a row until it is finally able to run when it should. With the above script this isn't happening because I am not making it reload until it queues correctly, but you can see that the pages will be often complete before the script even has a chance to run.If I pack the userscript in Chrome into a testing standalone addon, which runs-at document-start there are 0 issues like this, the script runs as soon as the page exists flawlessly, so Tampermonkey should be able too.
I can't tell if this is being caused because the Tampermonkey extension is not running as soon as it should whenever the pages open or if the userscript is not being injected as soon as it should due to some slowliness caused by Tampermonkey, but this behavior is very unreliable and is forcing me to stop considering supporting my scripts for Tampermonkey because I don't know what else I can do if the extension is not reliable enough to run the code when it should or reasonably close to it.
I have no issues with Greasemonkey nor when I convert the userscript into a Chrome extension, it's only with Tampermonkey that I am having these complications which are making the script unusuable.
Can you update the extension to be more reliable in this aspect? Preferably it should follow Greasemonkey's setting, which I quote:
Instead of Tampermonkey's current:
Also I am using Tampermonkey 3.10.84 and running just the one script, no other.