document.all willful violation

[annevk]

Other than some event loop mismatch which I think we'll be able to resolve more easily, HTML (and JavaScript engines) also contain(s) the document.all willful violation.

https://html.spec.whatwg.org/multipage/obsolete.html#dom-document-all

How much appetite is there to try to formalize this exception?

[erights]

@annevk Of all the outstanding web reality issues, this is one of the least appetizing. But like the others, if everyone's gonna do it and we can't kill it, we're better off trying to pin down the best precise semantics we can get away with.

I remember there was a subtle but important difference between how Firefox did this vs how either or both Safari and Chrome did this. I think I remember that the Firefox way seemed like a better first step towards a cleaner semantics which is still compat with web reality. But I no longer remember the details.

Anyone? @brendaneich , I think you were the one who explained this to me, ages ago. Does this ring a bell?

[erights]

I think I also remember that the FF way, if we had been able to agree on it in time, would have allowed us to banish the weirdness in strict mode. However, we were not, and so this was a missed opportunity for a significant cleanup.

Does any of this ring a bell? I think this was only discussed in side conversations, and so was probably never captured in the notes of the main mtg.

[domenic]

I'm not aware of any semantic differences between browsers on this feature, but I could be wrong...

My main conflict is: how would we spec this? Presumably we wouldn't have ES talk about a specific object called document.all. We'd instead define some kind of generic behavior (what V8 calls an "undetectable object") and then HTML could say "document.all is an undetectable object". But you'd never ever ever want people to use "undetectable object" in other cases.

I think with diligence we'd be able to prevent people from doing so (e.g. people might want to use it to deprecate features in the future, and we can tell them not to do that). But it still strikes me as weird to create this more generic interface that we insist must only be used once by one client. The current structure of one obviously-ugly willful violation seems more explicit about being a one-off, to me.

[bterlson]

I'm not convinced we want the spec machinery for document.all's exotic behavior in ECMA262 - it'd be a fair amound of machinery and wouldn't actually be used in any normative spec text (because we certainly wouldn't define document.all in 262, right?) I'm personally ok with the current level of specification here.

[allenwb]

If we think it was important to document, we could do so as an standalone Ecma Technical Note

[bterlson]

@allenwb do you think the formalization we would add in the technical note would be beneficial beyond what is already mentioned in HTML?

[allenwb]

@bterlson probably a question best answered by browser implementors. Do they think that the definition provided in the HTML spec is inadequate for somebody who is trying to create an interoperatable implementation? Is so, are the inadequacies of the sort that technically could be best addressed by a TC39 document?

[bterlson]

@annevk / @domenic: ^ ?

[domenic]

There have been no implementer complaints. This is more about us being concerned that we're bad citizens by monkeypatching ToBoolean/typeof/abstract equality comparison, which is "ES's territory".

[bterlson]

The grossness of the spec seems commensurate with the grossness of the desired semantics. I see no problems here :)

[annevk]

The other thing is that by not including it here you make it harder to reason about an implementation of the language and might miss something important. And indeed you miss opportunities such as disabling the functionality in certain contexts.

[ljharb]

We do have Object.prototype as an "exotic immutable prototype object" (pending a formal proposal to make that not be exotic), would we want to define an "exotic falsy object", so that the WHATWG spec could describe document.all as that? We'd then be able to forbid "exotic falsy objects" in any contexts that document.all didn't need to exist, as @annevk pointed out.

[bterlson]

@ljharb that's how I'd do it, probably, but I think the current spec text in HTML is straightforward (and unlikely to be broken by spec revisions) so I'm reluctant to add this machinery. Immutable prototype objects have a use case. Crazy undefined-seeming objects don't beyond this compat hack.

[evilpie]

So for the Firefox history see bug 792108. Basically we used to have a special way of passing along a "lookup context" (detecting flags) when calling [[Get]]. So for example when executing typeof document.all we would pass along the detecting flag and [[Get]] would return undefined in that case. Having special flags for property lookups is just totally incompatible with ES and was removed completely.

Oh and the current approach shared with all other? browsers is basically a lot simpler and just needs a check in a handful of places. (ToBoolean, typeof, ?)

[annevk]

FWIW, despite @bterlson being okay with HTML defining some of it, I would like TC39 to own up to the entirety of the JavaScript subsystem in browsers, including this bit.

I think whenever we uplifted such warts it has led to a greater shared understanding of the computing platform we offer to the world.

[erights]

@annevk , I would like this too, but not enough to put much time into it. If you'll take the lead, be the champion, write proposals, and take them to the committee, I could probably find the time to review.

@brabalan, @charguer, @totherme, @Mbodin, @sergiomaffeis, @edgemaster In order for any formal proof about a formal model of JavaScript to be relevant to real-world JavaScript, the differences between the formal model of JavaScript and real-world JavaScript need to be minimized and understood. As explained at https://github.com/FUDCo/proposal-frozen-realms/issues/4#issuecomment-207276873 (including the containing thread), the formal model can be of something much less than the full language and still have the proofs be relevant to actual JavaScript.

The key is that the formal model states invariants that are true for the whole language. Ideally, these invariants are themselves stated normatively in the spec, so any spec detail, say in Date, that violates these invariants creates a spec inconsistency, rather than invalidating the invariant. When there's no overriding issue to the contrary, we can expect the detail to be corrected to conform to the invariant, rather than the other way around. This leaves proofs that depend on those invariants intact.

This is why document.all is more important than, for example, getting the details of Date correct. We know document.all violates what would otherwise be invariants. Because everyone implements document.all, the "invariants" it violates are no longer invariants. They are at best "invariants but for one exception". This would not be the first. === is reflexive but for NaN. To know whether such an invariant violation can be leveraged into an attack requires one of the following two approaches:

• knowing precisely in what way the invariant is violated, so we can reason about it
• knowing we can deny any potential attackers access to the exceptional values -- those for which the invariants are suspended.

Fortunately for document.all, denial is a viable approach. Indeed, denial is the more attractive approach when the threat model allows it. (By contrast, no conceivable threat model for JS would allow denial of NaN.) Fortunately for the JS security work I've been doing --- Caja, SES, ocaps, https://github.com/FUDCo/proposal-frozen-realms , etc --- my threat model does allow reliable denial of document.all, and so my security work need not consider these as invariant violations. As far as my work in concerned, proofs relying on "invariants" broken only by document.all are still valid. Hence my lack of energy for this particular exception.

[evilpie]

This is basically everything https://github.com/evilpie/ecma262/commit/0514f06c765f34deebfca5b8dc4490dbf766a544 that is implemented differently by SpiderMonkey for undetectable objects.

[bterlson]

@evilpie looks mostly good but doesn't handle how to compare two undefined-like objects.

[allenwb]

If TC39 does anything to specify this behavior it should not be in the main body of the spec and it shouldn't use general purpose sounding language such as “undetectable object”. Much preferable would be something like “documentAllLegacyHack object” that makes it clear that the behavior exists for only one specific purpose.

I would also still prefer that it note be in ECMA-262 at all, not even Annex B. I still like the idea of a separate Ecma technical report that not only describes the spec. delta but also explains why it exists and exactly how and why it violates invariants.

[evilpie]

@bterlson Thanks. Actually that is handled by Step 1, they still have both have an object type. And indeed in Firefox, Chrome and Edge window.all == window.all is true.

[bterlson]

In that case its the same object reference, but what if there were two of these kinds of objects in existence?

[evilpie]

1. There won't be. (minus the cross-iframe case)
2. The code you proposed wouldn't actually be hit.
3. We only really care about people doing window.all == undefined or window.all == null.
4. Don't do that.

[erights]

@allenwb All my reasons for wanting this are served at least as well by a tech report. Altogether, I agree that the tech report is better than anywhere in the spec proper.

[bterlson]

@evilpie There are multiple of these but I now see that == returns false in these cases (eg. var f = document.createElement('iframe'); document.body.appendChild(f);f.contentWindow.document.all == document.all). Will fix your point 2.

[littledan]

If we write this in a spec, I'm all for writing down that it's an unfortunate legacy decision. What are the advantages to a separate technical report vs Annex B? It's already confusing as an implementer to have things separated out in Annex B, but at least it's in the same document and it's easy to link between them; using a separate technical note would just make things harder. Does anyone on this thread think that web browsers could unship document.all falsy behavior without breaking the web?

[bterlson]

Depends on how big of breakage :) But I haven't checked in a few years so perhaps things have improved since we were all forced to implement this?

The advantage of a TR is it doesn't cruft up 262 and ECMA owns the spec text. However the primary motivation for moving these semantics to 262 is discoverability and better spec factoring. I don't think either of these goals are achieved with a TR. I'd argue they're regressed. Personally, I think we should either have this in 262 or leave it in HTML.

[erights]

@littledan says:

Does anyone on this thread think that web browsers could unship document.all falsy behavior without breaking the web?

Browsers are not the only EcmaScript platform.

Putting it into Annex B implies that it is normative-optional. I would prefer some status weaker than that, in which case even Annex B is inappropriate.

Worse, Annex B implies that it is normative-mandatory for a platform that claims to be a web browser. This is definitely too strong. We should allow this one weirdness in this one special case. But we should never require it. If any browsers want to try killing it, we should not add to their costs for that brave experiment.

[domenic]

Worse, Annex B implies that it is normative-mandatory for a platform that claims to be a web browser. This is definitely too strong. We should allow this one weirdness in this one special case. But we should never require it. If any browsers want to try killing it, we should not add to their costs for that brave experiment.

I don't think the presence or absence of things in Annex B has any impact on browsers' willingness to experiment. It is certainly not a cost.

[bterlson]

Even with #673 (ie. implemented in the main spec text) there is no normative requirements for non-browser ECMAScript implementations because, as the note says, those objects are never created. It also doesn't say that document.all must be a ULEO, so browsers would be able to get rid of document.all without affecting its 262 conformance (although it would possibly regress HTML conformance in a way they are unlikely to care about assuming its web compatible).

[erights]

@domenic If it is normative-optional, then it allows a test to be added to test262 that first probes to see if the property is present. If not, then the test passes. Otherwise, the test262 test would test for conformance. Either we should add such tests to test262 or we should withdraw the Annex B statement that is features are normative-optional. Failing a valid test262 conformance test is indeed costly.

[erights]

@bterlson "ULEO"?

[bterlson]

@erights undefined-like exotic object (sorry, I just coined the term). document.all will never be specified in 262 so there will never be a test262 test for these semantics I would bet.

[domenic]

@erights I think where we are missing each other is that you think failing test262 prevents browsers from experimenting; that is not true as far as I know.

[erights]

Not "prevents". Just raises the costs.

[domenic]

Right, I also haven't seen any evidence of that.

[erights]

@bterlson I am certainly not willing to allow this into the main spec text, nor to allow this to appear in the specified state anywhere other than, specifically, "document.all". Our ability to censor its presence under certain threat models presumes we know where this anomalous value is.

[littledan]

@erights Can you elaborate on this threat model?

[erights]

@littledan For example, safer JavaScript plugins as in: https://drive.google.com/file/d/0Bw0VXJKBgYPMeFRjenpFb0dYNnM/view

[littledan]

What if we separated out documenting what is there right now, which this PR does, from changing what web browsers ship in new proposals?

[erights]

Say an initialize-time shim deletes document.all and wishes to claim that the resulting shimmed state is conformant to some version of the spec. Normative-optional would allow this. Normative-mandatory would not.

Say that, instead, the shim replaces document.all with something else, for example and accessor property that always throws on access. Even normative-optional would disallow the shim claiming to have left a compliant system state in its wake.

[erights]

@littledan I do not understand what your "what if" means. A non-normative documentation of what's there, including bugs, is a fine form of natural science but has nothing to do with the mission of tc39.

[littledan]

Other standards bodies, e.g., WHATWG and W3C, do try to document the reality of what is shipped in web browsers alongside proposing new features and modifying semantics of existing features. Does TC39 have an articulated mission written down somewhere?

[domenic]

Say an initialize-time shim deletes document.all and wishes to claim that the resulting shimmed state is conformant to some version of the spec. Normative-optional would allow this. Normative-mandatory would not.

Say that, instead, the shim replaces document.all with something else, for example and accessor property that always throws on access. Even normative-optional would disallow the shim claiming to have left a compliant system state in its wake.

This doesn't make any sense to me.

If ES defines "ULEO"s and HTML says document.all is an ULEO, then a shim that deletes document.all has no impact on the shimmed state's conformance to ES (no matter whether ULEOs are normative-optional or normative-mandatory). It only has an effect on the shimmed state's conformance to HTML (it is no longer conformant). Replacing it with a throwing accessor will similarly have no impact on the state's conformance to ES but will break the state's conformance with HTML.

The only way that such a shim would have any impact on ES conformance is if ES did something strange like "The initial value of the global this value's document getter's result's all getter's result must be an ULEO". Nobody is proposing putting that in, either as normative optional or normative mandatory.

[erights]

ES must not allow any ULEOs other than document.all itself. If we can't agree to limit it to that, then we're better off not admitting ULEOs into any tc39 documents at all.

[domenic]

How do you propose ES puts such a restriction on host environments? It seems outside of the scope of ES.

[erights]

Currently, ES restricts host environments from exposing anything with this behavior to JavaScript, period. It is in the scope of ES to state invariants that must be obeyed by all values exposed to JavaScript.

If we carve out an exception, we can say precisely that a value with ULEO behavior, if it exists at all, can only be found as the value of the "all" property of an object that is the value of the "document" property of the global object.

Although you meant it as a counter example, your text is an equally good starting point:

"The initial value of the global this value's document getter's result's all getter's result must be an ULEO"

[erights]

Sorry, although it is a starting point, your "must" must be replaced with "may". This must be the only exception allowed to the invariants that document.all violates.

[concavelenz]

Could we just leave it unspec'd for a few years and then just kill the special behavior?

On Aug 17, 2016 7:32 PM, "Mark S. Miller" notifications@github.com wrote:

Sorry, although it is a starting point, your "must" must be replaced with "may". This must be the only exception allowed to the invariants that document.all violates.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tc39/ecma262/issues/668#issuecomment-240607464, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMDKvTb6pV9rKXburriAJcejT8iGXxYks5qg8Q3gaJpZM4JkR6F .

[erights]

@concavelenz Possibly.

We have succeeded at killing even some bad old unspeced behaviors that were implemented compatibly across browsers. For example: the old de facto function.caller behavior on builtin functions. AFAIK, function.caller is still present on sloppy functions. And its absence on strict functions is not the killing of an old thing. But on builtins it was universally available, and we succeeded in killing it anyway.

This required extraordinary effort over many years. If we have reason to believe that we might eventually be able to kill document.all as well, that would be attractive. But killing builtin-function.caller required more than just inattention. We went through the whole poisoning phase to discourage use.

Does anyone have data on existing use of document.all? Is anyone prepared to take on a multiyear mission of draining this swamp from the web until we can kill it? Anyone have an adoptable interim mechanism to propose, analogous to poisoning, to discourage continued use of document.all? I know why I cared enough in the case of .caller, .callee, and .arguments, and several other historical misfeatures. However, why would anyone care enough about document.all? Unless someone takes on this mission, it won't happen.

[annevk]

Per https://www.chromestatus.com/metrics/feature/timeline/popularity/83 usage is at 5% (threshold is 0.03%) which means it's basically impossible to ever remove.

(And we tried neglecting it for many many years already. Didn't work.)