Symbol.private vs WeakMap semantics

[Igmat]

Originally I posted following messages in #149 answering to @erights concern about Membrane pattern implementation with Symbol.private approach. I've decided to create new issue here for several reasons:

1. Previous thread become too big to follow it
2. While this topic is related to original discussion it's actually important enough to be separated
3. 21 days passed from the moment I posted it, 16 from @erights promise to review it - and NO response so far (even though he was active in https://github.com/rdking/proposal-class-members/issues/10#issuecomment-445988975)
4. According to this: https://github.com/tc39/proposal-class-fields/issues/175#issuecomment-441698604, https://github.com/tc39/proposal-class-fields/issues/149#issuecomment-431527298 and September meeting it seems to be a major concern against Symbol.private
   • Unfortunately, there are no notes from latest meeting where it was discussed
   • there is also concern about lack of ergonomic syntax, but I already proposed few solutions (e.g. #149, #134 and https://github.com/tc39/proposal-class-fields/issues/149#issuecomment-442242427), and I'm not the only one who did it (e.g. https://github.com/rdking/proposal-class-members/issues/10), so IMO it could be solved if Symbol.private will reach stage 1, so we'll be able to work on it or create follow-on proposals

@ljharb, @erights, @littledan, @zenparsing could you, please answer to this:

1. Is Membrane pattern a major concern?
2. Did I address Membrane+Symbol.private issues with following solution?
3. Is lack of ergonomic syntax is also a major concern?
4. Should I summarize syntax solutions for Symbol.private (including my yet unpublished ideas)?

Copied from https://github.com/tc39/proposal-class-fields/issues/149#issuecomment-441057730

---

@erights, as I said before there is a solution that makes your test reachable using Symbol.private, here's a sample:

function isPrimitive(obj) {
    return obj === undefined
        || obj === null
        || typeof obj === 'boolean'
        || typeof obj === 'number'
        || typeof obj === 'string'
        || typeof obj === 'symbol'; // for simplicity let's treat symbols as primitives
}

function createWrapFn(originalsToProxies, proxiesToOriginals, unwrapFn) {
    // `privateHandlers` are special objects dedicated to keep invariants built
    // on top of exposing private symbols via public API
    // we also need one-to-one relation between `privateHandler` and `original`
    const privateHandlersOriginals = new WeakMap();
    // we're keep track of created handlers, so we'll be able to adjust them with
    // newly exposed private symbols
    const allHandlers = new Set();

    // just simple helper that creates getter/setter pair for specific
    // private symbol and object that gets through membrane
    function handlePrivate(handler, privateSymbol) {
        const original = privateHandlersOriginals.get(handler);
        Object.defineProperty(handler, privateSymbol, {
            get() {
                return wrap(original[privateSymbol]);
            },
            set(v) {
                original[privateSymbol] = unwrapFn(v);
            }
        })
    }

    function wrap(original) {
        // we don't need to wrap any primitive values
        if (isPrimitive(original)) return original;
        // we also don't need to wrap already wrapped values
        if (originalsToProxies.has(original)) return originalsToProxies.get(original);
        const privateHandler = {};
        privateHandlersOriginals.set(privateHandler, original);
        allHandlers.add(privateHandler);

        //       note that we don't use `original` here as proxy target
        //                      ↓↓↓↓↓↓↓↓↓↓↓↓↓↓
        const proxy = new Proxy(privateHandler, {
            apply(target, thisArg, argArray) {
                thisArg = unwrapFn(thisArg);
                for (let i = 0; i < argArray; i++) {
                    if (!isPrimitive(argArray[i])) {
                        argArray[i] = unwrapFn(argArray[i]);
                    }
                }

                //          but we use `original` here instead of `target`
                //                           ↓↓↓↓↓↓↓↓
                const retval = Reflect.apply(original, thisArg, argArray);

                // in case when private symbols is exposed via some part of public API
                // we have to add such symbol to all possible targets where it could appear
                if (typeof retval === 'symbol' && retval.private) {
                    allHandlers.forEach(handler => handlePrivate(handler, retval));
                }

                return wrap(retval);
            },
            get(target, p, receiver) {
                receiver = unwrapFn(receiver);
                //       but we use `original` here instead of `target`
                //                         ↓↓↓↓↓↓↓↓
                const retval = Reflect.get(original, p, receiver);

                // in case when private symbols is exposed via some part of public API
                // we have to add such symbol to all possible targets where it could appear
                if (typeof retval === 'symbol' && retval.private) {
                    allHandlers.forEach(handler => handlePrivate(handler, retval));
                }

                return wrap(retval);
            },
            // following methods also should be implemented,
            // but it they are skipped for simplicity
            getPrototypeOf(target) { },
            setPrototypeOf(target, v) { },
            isExtensible(target) { },
            preventExtensions(target) { },
            getOwnPropertyDescriptor(target, p) { },
            has(target, p) { },
            set(target, p, value, receiver) { },
            deleteProperty(target, p) { },
            defineProperty(target, p, attributes) { },
            enumerate(target) { },
            ownKeys(target) { },
            construct(target, argArray, newTarget) { },
        });

        originalsToProxies.set(original, proxy);
        proxiesToOriginals.set(proxy, original);

        return proxy;
    }

    return wrap;
}

function membrane(obj) {
    const originalProxies = new WeakMap();
    const originalTargets = new WeakMap();
    const outerProxies = new WeakMap();

    const wrap = createWrapFn(originalProxies, originalTargets, unwrap);
    const wrapOuter = createWrapFn(outerProxies, originalProxies, wrap)

    function unwrap(proxy) {
        return originalTargets.has(proxy)
            ? originalTargets.get(proxy)
            : wrapOuter(proxy);
    }

    return wrap(obj);
}

const privateSymbol = Symbol.private();
const Left = {
    base: {
        [privateSymbol]: ''
    },
    value: '',
    field: privateSymbol,
};
const Right = membrane(Left);

const { base: bT, field: fT, value: vT } = Left;
const { base: bP, field: fP, value: vP } = Right;

// # set on left side of membrane
// ## set using left side field name
bT[fT] = vT;
assert(bP[fP] === vP);

bT[fT] = vP;
assert(bP[fP] === vT);

// ## set using right side field name
bT[fP] = vT;
assert(bP[fT] === vP);

bT[fP] = vP;
assert(bP[fT] === vT);

// # set on right side of membrane
// ## set using left side field name
bP[fT] = vT;
assert(bT[fP] === vP);

bP[fT] = vP;
assert(bT[fP] === vT);

// ## set using right side field name
bP[fP] = vT;
assert(bT[fT] === vP);

bP[fP] = vP;
assert(bT[fT] === vT);

Copied from https://github.com/tc39/proposal-class-fields/issues/149#issuecomment-441075202

---

I skipped fP->fT and fT->fP transformations in previous example for simplicity. But I'll mention such transformation here, since they are really easy handled when private symbols crosses boundary using public API of membraned object

set on left side of membrane

set using left side field name

bT[fT] = vT;

goes as usual, since happens on one (left) side of membrane

assert(bP[fP] === vP);

• since fP is private symbol, get called on proxy target
• which is special privateHandler object
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP

• assertion is TRUE

  ---

bT[fT] = vP;

• unwrap vP to vT
• set vT to bt[fT] as usual

  assert(bP[fP] === vT);

• proxy skipped to target (privateHandler)
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP
• wrap vT to vP
• assertion is TRUE

set using right side field name

bT[fP] = vT;

• unwrap fP to fT
• set vT to bt[fT] as usual

  assert(bP[fT] === vP);

• wrap fT to fP
• proxy skipped to target (privateHandler)
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP

• assertion is TRUE

  ---

bT[fP] = vP;

• unwrap fP to fT
• unwrap vP to vT
• set vT to bt[fT] as usual

  assert(bP[fT] === vT);

• wrap fT to fP
• proxy skipped to target (privateHandler)
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP
• wrap vT to vP
• assertion is TRUE

set on right side of membrane

set using left side field name

bP[fT] = vT;

• wrap fT to fP
• wrap vT to vP
• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

    assert(bT[fP] === vP);

• unwrap fP to fT
• get bT[fT] which equals vT
• unwrap vP to vT

• assertion is TRUE

  ---

bP[fT] = vP;

• wrap fT to fP
• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

    assert(bT[fP] === vT);

• unwrap fP to fT
• get bT[fT] which equals vT
• assertion is TRUE

set using right side field name

bP[fP] = vT;

• wrap vT to vP
• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

    assert(bT[fT] === vP);

• get bT[fT] which equals vT
• unwrap vP to vT

• assertion is TRUE

  ---

bP[fP] = vP;

• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

    assert(bT[fT] === vT);

• get bT[fT] which equals vT
• assertion is TRUE

[jridgewell]

Is Membrane pattern a major concern?

Yes. Proposals will not gain consensus if they break membranes.

Did I address Membrane+Symbol.private issues with following solution?

Not all of them, see below.

Is lack of ergonomic syntax is also a major concern?

"Ergonomic syntax" meaning something like obj.#priv instead of obj[priv]? Then yes, this was a concern at the Sept meeting. It'll be hard to get everyone to agree without a specific syntax.

Should I summarize syntax solutions for Symbol.private (including my yet unpublished ideas)?

I think most of the committee likes .#. Anything that uses private for declaration and not # will be nearly impossible.

---

Membrane stuff:

• bT[fT] = vP;
  • You're first assumption is that vP is immediately unwrapped to vT. This is impossible, there's not proxy interaction to do the unwrapping.
• bT[fP] = vT;
  • Again, you're unwrapping as the first step which is impossible.

There's a few more like this, but you're getting close.

A big issue is all your proxy handlers will fail "has" checks, since you're using defineProperty, even if the original value doesn't actually have that private symbol. What's needed isn't installing these hooks onto the shadow target, but a way for proxies to learn of private symbols and then begin trapping them.

[Igmat]

@jridgewell thank you for response. I guess it's a good start for constructive discussion.

1. • bT[fT] = vP;

     • You're first assumption is that vP is immediately unwrapped to vT. This is impossible, there's not proxy interaction to do the unwrapping.

   • bT[fP] = vT;

     • Again, you're unwrapping as the first step which is impossible.

   You probably missed the main point - it's actually double membrane (one for code we are wrapping, second for everything outer). There is always a Proxy that does wrap/unwrap operations.

2. A big issue is all your proxy handlers will fail "has" checks

   Thanks for pointing this out - you inspired me to use Object.getOwnPropertyDescriptor which may also lead to avoiding unneseccary memory and time consumptions.

3. It'll be hard to get everyone to agree without a specific syntax.

   Should I treat this answer as yes to original question?

I also just get an idea how to create runnable tests that emulates Symbol.private for Membrane implementation. I'll publish small repo tomorrow with it, so we could have more material for investigation.

[erights]

@Igmat thanks for creating a self-contained issue. I will respond here. I would like to indicate by when, but I've already exceeded previous expectations --- my apologies.

[Igmat]

@jridgewell, @erights I just realized that I accidentally broke layout of my answer in https://github.com/tc39/proposal-class-fields/issues/183#issuecomment-447122079. I updated the comment, so my points become visible - you probably may review it since those answers could be helpful for understanding.

[jridgewell]

it's actually double membrane (one for code we are wrapping, second for everything outer). There is always a Proxy that does wrap/unwrap operations.

Ahh, I didn't see that. I'm not sure how Mark feels about it, but that's not the membrane design as I understand it. On the left side and right sides, values shouldn't be wrapped. Only a left-sided object pulled through to the right side should be wrapped (and the other direction too).

you inspired me to use Object.getOwnPropertyDescriptor which may also lead to avoiding unneseccary memory and time consumptions.

I'm not sure I follow. The code I'm thinking of is fT in bP, which would go directly to the shadow target. Since there's a getter/setter there, it'd return true, even if the original object does not have a value on fT.

Should I treat this answer as yes to original question?

Yah, I have a feeling it's like a 90% requirement. Not strictly necessary, but the committee likes it enough that it must be a very, very good reason not to include it.

[hax]

I think most of the committee likes .#.

I very doubt this. There is a big difference between "I like it" and "ok, I accept it".

Anything that uses private for declaration and not # will be nearly impossible.

While I understand why private keyword has issues, I just don't understand why no use of # is impossible. For example classes 1.1 do not use #. The reason of rejection I heard is "no public field" not "no #".

[ljharb]

There were a number of reasons it was rejected; no public fields (and no possibility of them) was certainly a big one.

[hax]

@ljharb This is why @rdking working on a new proposal which add public field. The point is if he or anyone finish a proposal, will it be consider? Or just arbitrarily reject it with some unknown reason? As current responses in this repo, I'm very worry about that. So I hope you or someone can just tell us all the reasons.

[jridgewell]

Let's keep this on the topic of Membranes and Private Symbols.

[Igmat]

Sorry for the delay (Christmas and New Year holidays takes too much preparation =/), but continuing the topic.

As I promised, here a little test project for Membrane pattern with Symbol.private: https://github.com/Igmat/membrane-test-for-symbol-private

Important notes:

1. Since there are no Symbol.private in any environment, I treated all Symbols as private
2. This file implements Proxy replacement emulating how Proxy will interact with Symbol.private
   • Unfortunately, it seems to be limited, because it may not implement my vision of tunneling as I described in https://github.com/tc39/proposal-class-fields/issues/158#issuecomment-432809666 in So why do we need tunneling? section
   • Most probably, it works like @rdking described in his https://github.com/tc39/proposal-class-fields/issues/158#issuecomment-432823513
   • From another hand, both previous statements could be wrong, since I didn't have enough time to test these assumptions
   • But important thing is that it actually has NO EFFECT on discussed problem of Membrane and Symbol.private issue
3. I made few minor fixes comparing to original post (proper proxy for function and correct usage of argArray)
4. No additional setup, just run npm test even without installing packages (I've used only node's native API)
5. If you find ANY mistakes or new important edge cases that isn't covered in tests, feel free to comment them here, or via issues/PR's in my repo

Next steps:

1. I, probably, can create more real-world like test-cases if needed. What do you think?
2. I haven't checked my idea about work-around for @jridgewell issue about has check. Is it really important? I realize that it could break some invariants, but at first glance it looks very minor problem.
3. Do you need better infrastructure around it (test-runner, compiler, readme, etc.)?
4. I'll create separate issue for Symbol.private syntax later, since it's a deal-breaker

@erights, @jridgewell, @jridgewell, @littledan does it help? Do you have any comments about it?

[rdking]

@Igmat Take a look here. I took a shot at a rough implementation of Symbol.private in ES6. See if you can't implement your testing with that.

[Igmat]

@erights, @jridgewell, @littledan I updated my Membrane implementation, so now it passes all tests.

Big thanks to @jridgewell with his help explaining some points I missed and providing better test suites.

[littledan]

I am sorry about the delay in these reviews. I want to just point out that a single slide might include constraints that are considered important, but might not be exhaustive, so an implementation passing certain tests might not be sufficient.

I think we had some pretty specific reasons for each aspect of the WeakMap semantics in this proposal. I wrote some notes about it in https://gist.github.com/littledan/f4f6b698813f813ee64a6fb271173fc8 . This might turn into a TC39 presentation if private symbols semantics are re-raised to the committee.

[rdking]

@littledan Would it be fair to say that the document you've got at that link is at least somewhat representative of TC39's desires for a private data proposal in general?

[littledan]

I am not claiming that. (How could I make such a general claim when @jridgewell has said he will advocate further for private symbols?) This is a there-is rather than for-all style claim: there exist TC39 delegates who have advocated for these design goals, and I haven't heard anything from them about changing their mind due to recent discussion. TC39 consensus would require persuading everyone.

[rdking]

Given the "consensus = absence of veto" way that TC39 handles things, persuading everyone doesn't really seem to be a requirement. The mere fact that there are TC39 members who are still vocal about not supporting the collection of features that is class-fields and yet not willing to veto this proposal due to presumably political reasons is enough to tell me that there is no true "consensus", only passive absence of disagreement at best. Please stop conflating the two.

When I raised the question, I wasn't asking if the document represented the general subset of wants that were held by all members, but rather that the document represented a general collection of wants held by various members. I can't bring myself to believe that you all are on the same page with regards to this proposal given that even TC39 members are coming up with counter-proposals.

All that having been said, I would like to hear from TC39 about proposal-class-members, even though I'm fairly certain that none of you have any intention on halting the progress of the current proposal. Should class-members in its present form have been presented along side class-fields in its present form (but as stage 1 proposals), I would like to know how class-members would have faired.

[littledan]

@rdking For better or worse, the way TC39 works is, we'd need the consensus to advance a stage, as well as the consensus to move a feature back a stage.

[rdking]

@littledan My point is that your wording (saying "consensus" all the time) is ridiculously misleading. It presents the image that TC39 took a vote, and the majority agreed, when instead what actually happened is a question like "Does anyone think we should abandon this?" was asked, and no one answered. The difference here is critical. Very few people have the intestinal fortitude to stand up against a crowd or be the loner in a room full of cliques. Do you honestly believe that if the board were gathered in whole, the ups and downs of the proposal were thoroughly explained, and a true ballot vote was held, that there would be a "consensus" to proceed with this proposal?

[bakkot]

when instead what actually happened is a question like "Does anyone think we should abandon this?" was asked, and no one answered

No, what happened was we asked "do we have consensus to advance to stage X?" and we did in fact have consensus (after many, many, many hours of discussion) at each stage. A "no" vote at any stage past 1 would not mean "abandon", it would mean "there's still something I consider to be a serious issue I think should be worked on more". It is quite common for proposals to fail to get consensus.

Do you honestly believe that if the board were gathered in whole, the ups and downs of the proposal were thoroughly explained, and a true ballot vote was held, that there would be a "consensus" to proceed with this proposal?

The committee was gathered in whole, the ups and downs of the proposal were thoroughly explained, and then we asked if there was consensus - if everyone was OK with the proposal in its current state. And we had consensus. That's how the process already works.

[rdking]

@bakkot

No, what happened was we asked "do we have consensus to advance to stage X?"

I didn't know the exact question. However, does this question need to be followed by everyone voting "yes"? A majority voting "yes"? Or can most simply be silent with only 1 or 2 actual "yes" votes, as long as there are no "no" votes? All of us outsiders would like to better understand the nature of this "consensus".

Even knowing this much will likely help us have more confidence in the current process.

[bakkot]

It's not a formal vote; "consensus" is deliberately somewhat vague. In practice, though, if at least one person thinks there are remaining issues that should be resolved before advancing, or otherwise thinks a proposal should not advance at this time, and says so, the proposal will not advance.

Of course, that doesn't mean everyone is in agreement with all particulars. Any complex design will have points of disagreement, particularly in a group as large and varied as ours; design is an exercise in compromise. So what generally happens is that we discuss things to death until people most people agree on one design, which is adopted; eventually people who still disagree decide that it's unlikely that they'll persuade people to their side, and hence that the choice is between the feature never going forward or going forward despite their disagreement on whatever point, and then they need to make that call. This, for example, is what I understand to have happened on the question of Define vs. Set, and how it came to pass that we have consensus on the current proposal despite some committee members continue to favor Set.

[hax]

@bakkot What you said just prove my opinion: process failure. "consensus in the room" works for normal issue, but doomed on the controversial issues. It just become a game that who give up first. Some time a delegate just happened miss a f2f meeting, and you treat it as "consensus"... And if you check the meeting notes, you will find there are some people who disagree just disappear in the later meeting... Fine.

And this broken proposal have several such controversial issues!

[rdking]

@bakkot Thank you. That was one of the best and clearest answers about the process we've received since I jumped into this fray.

Now, can you give a similar explanation on why other candidate proposals cannot be used as a reason to unseat a stage 3 proposal, especially if one of the other candidates offers less trade-offs while still satisfying the primary goal?

[ljharb]

Because everyone would have to come to consensus to “unseat” anything - if a single person remains convinced that the current proposal should remain stage 3, and that no other competing proposal should advance, then nothing will change.

[rdking]

@ljharb Isn't that a matter of how the question is asked? The meaning of consensus and veto, given the way @bakkot described the process, varies greatly with the way the voting question is asked.

In the case of a competing proposal, of course the incumbent proposal's champion is likely to veto. Is that champion allowed to block competition on such grounds?

Is such a consensus required before a proposal is ever even reviewed?

There are so many things that are not understood about TC39's process, and so many places where there is absurd amounts of room for improvement. But I'm willing to bet that making such improvements in the process is also subject to the existing process. If that's true, then there may be those in TC39 who have a vested interest in insuring that the process doesn't change.

... Maybe this is what @littledan meant when he told me that "the solution space is so small that there likely is no other viable possibility".

[ljharb]

Yes, they are allowed to block on those grounds; anyone can block on any grounds.

[hax]

@ljharb

Yes, they are allowed to block on those grounds; anyone can block on any grounds.

Anyone can refuse alternative proposal and insist on forwarding current stage 3 proposal. So those who think current proposal is broken have no other choice --- accept it (by leaving the room and be silent), or veto it totally which have very large social cost in the committee.

Isn't that a problematic process ??

[littledan]

I agree that we should get broader input on proposals than consensus in the room, and we should take this input seriously. But I don't think we will ever get 100% support from everyone outside of the committee for a proposal. I hope we can collect community feedback earlier in TC39's process so it can feed into what goes into committee decision-making for Stage 2 and 3. During this proposal's development, we did talk to several framework and transpiler community leaders, as well as a number of ordinary JavaScript programmers, but we can always improve the depth here, and I intend to work towards that.

[hax]

But I don't think we will ever get 100% support from everyone outside of the committee for a proposal.

I never think a proposal should get 100% support. But you at least should not get 50%+ disagree. Unfortunately you will never get 50%+ programmers support if you told the programmers all darkside of this proposal. Both @Igmat and me already gave you the feedback from the community.

[Igmat]

While I think that process and feedback analyzing is very important issue that worth discussing, but let's keep this thread for Symbol.private vs WeakMap semantics.

---

Could somebody mark all unrelated comments (including this one) as off-topic? @littledan, do you have such permissions?

[Igmat]

I finally finished Membrane implementation which works with Symbol.private and preserves all needed invariants (both described by @erights in his presentation and has checks raised by @jridgewell). There are still few places where adjustments are needed - but they are very small and share concepts I've used for other parts, so I just marked them with comments. You may check my implementation and additional tests for has checks in https://github.com/Igmat/membrane-test-for-symbol-private/pull/7.

In short, 2 things that made this possible:

1. Trick with adding all private symbols to all shadow targets when firsts are exposed + using double membrane concept (or should we call it two-side membrane?), forced such membrane implementation to preserve get/set invariants listed by @erights.
2. Small mutation of Object/Reflect global API's forced such membrane implementation to preserve has check invariants raised by @jridgewell.

And second point isn't hack (as it could look like at first glance), because ANY membrane implementation (with or without Symbol.private) REQUIRE mutation of these API's (at least by freezing them, but getter/setter pair will serve such purpose better), because otherwise malicious code will be able to get unwrapped originals by mutating this API's.

Actually problem is little bit deeper, because there are other global API's that could be mutated by malicious code in a way which will grant access to originals for it. So proper Membrane (with or without use of Symbol.private) inevitably becomes Realm (as in this proposal).

Also, I have to point out that whitelisting Proxy allows much simpler and cleaner solution for Membrane issue, but it doesn't necessary to provide this additional capability right now, instead it could be a separate proposal for future ES adjustments. You could check Membrane implementation with such Proxies in @jridgewell's fork.

From this point I can claim that Membrane ISN'T an issue for Symbol.private. Yeah, it becomes a little bit more complicated (while this pattern is very complicated itself even without Symbol.private), but it's still achievable - and I proved it (and what could prove the point better than working code that passes test?). Unless you find any other issues (and feel free to raise them here, or in test repo), we have to consider that one of 2 major issues with Symbol.private where solved.

Does anybody (@littledan, @erights, @ljharb, @jridgewell, @zenparsing) has any objections?

P.S.

Second major point is lack of ergonomic syntax, I'll cover it in another thread.

P.P.S.

To be honest, I expected much more involvement into discussing Membrane pattern implementation with Symbol.private (especially from @erights) since it was major argument against it, but it seems that nobody from committee (except @jridgewell) checked my arguments and/or implementation even though previously raised Membrane issue as deal-breaker which makes Symbol.private a non-starter solution.

[littledan]

Hid the process discussion as off-topic per @Igmat 's request. In general, this repository is focused on technical issues, not process.

[erights]

I expected much more involvement into discussing Membrane pattern implementation with Symbol.private (especially from @erights) since it was major argument against it,

Hi @Igmat , I expected so too. My delay is due to trying to first find the time to actually absorb your code, to understand it well enough to respond to it. Rather than continuing to postpone, perhaps it's better if I return to the incremental question approach that worked so well earlier in the thread.

(I would also like to acknowledge that your efforts on this issue, and the efforts of many people on these threads, greatly exceed my own. I do appreciate this. Your efforts are serious and careful, and so deserve more of my attention. But regardless, please help me use the limited attention I do have as well as possible. Thanks!)

I'll start with the biggest issue from the earlier q/a https://github.com/tc39/proposal-class-fields/issues/158#issuecomment-450598146 that I don't know how you'd solve. What does the following code do?

Reflect.getOwnPropertyDescriptor(rMembrane, x);

The current stage 3 proposal gives the same answer here as does the WeakMap model of private state: x is in no sense a property of the rMembrane proxy, or its shadow, or its real target. Rather, the real target of the rMembrane proxy is a key of the WeakMap(-like object), which is the real target of the x proxy. Thus, it seems that the rMembrane oroxy is a key in the x WeakMap proxy. Thus, this call returns undefined.

[erights]

I should point out that much of the reason I became excited in #158 about PNLOWNSF (PrivateName-like object with name-side faulting) is my hope that it would preserve the essence of the WeakMap model of private state: that the state is in the name, associating instances with values, rather than the state being in the instance.

In that thread, once we switched from private symbols to PNLOWNSF, we never did get back to the question I re-ask above. I would hope it would give the same answer: undefined, since the state is not in the instance. The only thing the instance contributes is its identity, which cannot trigger any behavior on the instance side.

[Igmat]

If I understand your question properly, this call will return undefined only if x isn't a property of rMembrane. To be a little bit more precise, it'll work in following way. Let's assume that x is Symbol.private, because otherwise it works as usual. Since we're using rMembrane, operation is held on the Left side. So execution flow:

1. x belongs to Right
   • this means that x passed the Membrane boundary
   • which means that x was stored as exposedSymbols
   • augmented Reflect.getOwnPropertyDescriptor checked that exposedSymbols has x and returns descriptor of real target and not shadow target (it could be wrapped if needed, but current implementation doesn't do it)
2. x belongs to Left
   1. x wasn't exposed to Membrane
      • this means that property was defined by left side
      • since Symbol.private bypasses Proxy, it was applied directly to shadow target
      • bypassing Proxy also means that this getOwnPropertyDescriptor will be applied directly to shadow target
      • which means that we'll get exactly same descriptor as was used for defining the property
   2. x was exposed to Membrane
      • at the time of exposure all shadow targets were checked for existance of such property
      • if such property existed at shadow target, it was also added to original object
      • all shadow targets changed a little bit their property descriptor (if it was there), to keep tracking it and properly applying changes to original target
      • since Reflect.getOwnPropertyDescriptor is aware of exposedSymbols it returns proper descriptor (and againg it could be wrapped if needed).

So if x doesn't exist as property (wasn't installed by any means) at any particular membraned object, it'll return undefined. But if it was added to membraned object it will return proper descriptor.

[erights]

This augmenting of Reflect.getOwnPropertyDescriptor is in the has patch at https://github.com/Igmat/membrane-test-for-symbol-private/pull/7/files#diff-168726dbe96b3ce427e7fedce31bb0bcR319 . Is this the one I should be looking at? I didn't find it in trunk but I didn't check other patches or branches.

[Igmat]

Yes, it is has patch I was referring to.

It isn't merged yet, since I wanted a little bit more attention to it from others like you, @erights, and @jridgewell, who helped me a lot.

[erights]

Thanks. So I see this code enhances Reflect.getOwnPropertydescriptor (hereafter Reflect.g22r ;) ) to capture mutable state, and to have mutable behavior based on this mutable state. However, I understand this is a shim for something that would be proposed, where the proposal may have better behavior than can be achieved by a shim. So, what would you expect to propose Reflect.g22r do?

As background, one of the design constraints we've been following is not to introduce new hidden state into the primordials. In an environment in which all the primordials are frozen, two subgraphs sharing only access to the primordials should not thereby have a communications channel, or any ability to cause or sense effects outside themselves.

There are four grandfathered-in exceptions to that:

• Date.now() and new Date() provide the ability to sense the current time, and thereby to sense duration. This opens side channels. See https://rawgit.com/Agoric/SES/master/demo/
• Math.random. Since it does not reveal the seed nor allow the seed to be set, if it were a cryptographically strong pseudo-random generator, which the spec allows but does not require, then it would not be a communications channel. But it still violates "no hidden mutable state".
• Current locale and current timezone. These can only be sensed, not changed, from JS. They are sufficiently stable in practice that it is hard to imagine them opening a side channel.

There are several new proposals that do introduce special powers, such as error stacks and weak references. That's why these proposals quarantine them in a separate System namespace that can be separately censored or virtualized. (Note that System as a mechanism is controversial, but some quarantining mechanism is needed for these.)

Reflect currently has no special powers or hidden state. We must uphold that.

The more subtle implication is that, if two graphs that are otherwise isolated are introduced to each other through some designed mechanism, the designer of this mechanism must be able to reason about how it enables those two graphs to communicate.

[Igmat]

My implementation was based on usual environment assumptions: global API that we have in browser/node which isn't frozen. I haven't tried to implement Membrane in frozen context, and therefore haven't think in such way. I always though that it's Membrane's responsibility to freeze all globals (which makes it a Realm), otherwise it doesn't fully protect neither side.

Symbol.private proposal actually doesn't add any hidden state to globals, but particular membrane does it, so we still uphold that Reflect has no special powers or hidden state.

[ljharb]

(note that a Realm doesn't have frozen globals initially unless it's a "frozen realm")

[erights]

Hi @Igmat , freezing primordials is actually orthogonal from membranes. https://github.com/Agoric/SES freezes the primordials but has no membranes. https://github.com/ajvincent/es-membrane is a membrane library that is perfectly usable (and used) outside of SES. Not quite orthogonal, in that there is a synergy between them. Within SES, membranes become an even more powerful abstraction mechanism for security patterns.

So imagine taking this membrane demonstration in two steps:

• What is the proposed change to the language itself? This can be approximated by a shim, but the proposal can be principled in ways that cannot be shimmed.
• Assuming a JS operating according to your proposed spec, how would you build a membrane library which could work in that JS if all the primordials were frozen before the membrane library got to load?

I'm breaking it up this way so that your demonstration can replace primordials, including Reflect.g22r, but only in order order to shim an approximation of proposed behavior. The behavior of the shim might have hidden mutable state, and be abusable as a communications channel, but only because the shim was not able to perfectly emulate the behavior you want to spec.

When I saw that override of Reflect.g22r, I first tried to understand it as if it were trying to perform such shimming.

Btw, apologies as well for perpetually moving the goal posts. The reason is that we've accumulated a large set of design constraints that are often inarticulate. That is, until we see something that would violate them, provoking us to articulate them. We will try to write down more of these over time. But it is inevitable that any large social technical design process will still have many of these, despite our best efforts.

[erights]

(note that a Realm doesn't have frozen globals initially unless it's a "frozen realm")

Right. But the constraint is that there's no hidden state, so freezing everything results in deep immutability, modulo the grandfathered exceptions. This is currently true everywhere.

Sometimes history helps explain the nature of the design constraint:

"No other hidden primordial state" was supposed to be true for ES5, and was almost true, except for one bit of hidden state that escaped our notice, and caused a security hole and vulnerability disclosure in Caja. In ES5, Date.prototype was an exotic Date object, in that it has the mutable internal slot that the builtin Date methods can read and write. For all builtin class-like abstractions F, F.prototype was an exotic F object. But Date was the only exploitable one.

Starting in ES6, for all builtin class-like abstractions F, F.prototype is now a plain object, except for the grandfathered exceptions, Function, Array and Number, none of whom have hidden mutable state. We made these exceptions one at a time as we found web-breaking cases caused by the new rule.

[Igmat]

Ok, I understand your point. In this case, there is only one full solution: whitelisting Proxy should I elaborate what is it? If you're intersted there is a @jridgewell's fork and I believe we may improve it to showcase everything important about it. Such Proxy will allow creating proper Membrane in frozen context.

Meanwhile, my implementation feels the gap between usual and whitelisting Proxy, when second isn't available and globals are mutable. So, you're partially right that it's a shim.

So, I propose to solve problem in two following steps:

1. Move forward with Symbol.private (which will grant users the possibility to have encapsulated state)
2. Adjust Proxy API with additional third argument - whitelist (Set of private symbols, which are trapped by that particular proxy, while all other Private Symbols are directly applied to proxy target and not captured by its traps).

Between step 1 and 2, we could use my existing implementation of a Membrane if we're able to create it before freezing globals.

P.S.

My implementation still covers all invariants from your presentation (as you can see in master branch), but fails some of has checks if we're talking about using it in frozen context.

[erights]

Such whitelisting is a painful degree of prior coordination that is absent from all the WeakMap-like approaches:

• WeakMaps themselves
• The stage 3 proposal
• The stage 3 proposal + decorators reifying PrivateNames
• PNLSOWNSF (I got the acronym wrong above. It is PrivateName-like special objects with name-side faulting.)

What do you mean by "move forward"? If you mean, keep exploring and problem solving here as you've been doing, that would be great. I've already learned several surprising things, and I'm sure such an exploration will teach more valuable lessons.

Has PNLSOWNSF been disqualified on other grounds? Frankly, it is the most private-symbol-like approach that I find plausible. I do not expect names reified as symbols, nor instance-side faulting, to work out. But, after these discussions, I no longer think it is impossible! Nevertheless, I would be more interested in discussing PNLSOWNSF as long as it remains promising.

[Igmat]

painful degree of prior coordination

what do you mean by that?

[erights]

You have to know what these are up front when you create the membrane. Or did I misunderstand?

[jridgewell]

What does the following code do? Reflect.getOwnPropertyDescriptor(rMembrane, x);

With transparent proxies, it would would retrieve rMembrane's target's x descriptor, without invoking any traps on the rMembrane proxy.

I would hope it would give the same answer: undefined, since the state is not in the instance. The only thing the instance contributes is its identity, which cannot trigger any behavior on the instance side.

I believe you're referencing the relationships presentation here, where the PrivateName-like object is a map the relates the instance to the value, where as symbols are a primitive that the instances relates to the value.

A map interface is certainly one approach to this, but I don't think it's the best. It's easiest to think of when we look at regular private state access:

class Ex {
  #x = 1;

  method() {
    return this.#x;
  }
}

In syntax, we use instance.#field which is very similar to instance.field, and instance.field is a primitive key relationship. That makes me think we should be designing instance.#field as a primitive key relationship.

This augmenting of Reflect.getOwnPropertyDescriptor is in the has patch at https://github.com/Igmat/membrane-test-for-symbol-private/pull/7 I understand this is a shim for something that would be proposed, where the proposal may have better behavior than can be achieved by a shim. So, what would you expect to propose Reflect.g22r do?

I would like to stop discussion on https://github.com/Igmat/membrane-test-for-symbol-private/pull/7, if we can. It'll just get us sidetracked.

I don't think there is an achievable end-goal for Reflect.g22r with this code. This path requires the membrane implementation to patch Reflect methods to keep its own state, it's not something Reflect would be able to do by default.

---

Instead, I'd like us to consider https://github.com/jridgewell/membrane-test-for-symbol-private the canonical repo for membrane/private-symbols. It's forked off @Igmat's initial code (and truly a huge thanks to him for starting with it!), and it's what I'll be presenting on during the private symbols discussion in the next meeting.

My design comes in stages:

1. First, there is only syntax.
   • Proxies are made transparent in unison.
2. Second, we introduce reification (deferred until Decorators, because they're not strictly needed)
   • Proxies gain a "whitelist" API for known reified private symbols

Now, to answer your last questions:

What is the proposed change to the language itself?

First, we introduce "private symbols".

These are an eventual reification for the primitive key in instance.#field. What I'll be proposing is that the reification is not allowed, yet, it'll be down the road. For now, there is only current class-fields syntax:

class Ex {
  #x = 1;

  method() {
    return this.#x;
  }
}

#x here is a non-reified private symbol. It behaves just like a regular symbol with prototype semantics. Eventually, you'd be able to create a reified private symbol:

const x = Symbol.private('x');
class Ex {
  [x] = 1;

  method() {
    return this[x];
  }
}

The syntatic is equivalent to the reified form in semantics (if we ignore that Symbol.private could be patched, we can discuss that at another point).

Again, I don't want to propose reification yet, only the syntax would be usable for now. There're merely shimmed in my repo because it's the only way to demonstrate all of this -- short of building an engine that would support the syntax.

Next, we have "transparent proxies".

(Ignore whitelist for now, I'll explain that next).

This is a change in the Proxy API that prevents any trap that receives a key (eg, get, set, g22r) from being called if the key is a "private symbol" (either syntatic use, or if it's eventually reified).

class Ex {
  #x = 1;

  constructor() {
    // Pretend there are proxy traps in the handler:
    const proxy = new Proxy(this, {});

    // This does not fire any proxy traps.
    proxy.#x = proxy.#x + 1;
    assert(this.#x === 2);

    // Now pretend `x` is the reified private symbol for `#x`:
    // This still doesn't fire a trap.
    proxy[x] = proxy[x] + 1;
    assert(this.#x === 3);

    // Also doesn't fire a trap
    // returns a { value: 3 } data descriptor
    Object.getOwnPropertyDescriptor(proxy, x);
  }
}

Finally, we have "proxy whitelist".

This (really not finalized design) is a mutable list of known, reified private symbols. If the whitelist contains a private symbol, than the Proxy's key-receiving-proxy-trap will be fired. Symbols can be added to the whitelist after creation of the proxy when the become known.

class Ex {
  #x = 1;
  #y = 2;

  constructor() {
    // Now pretend `x` is the reified private symbol for `#x`,
    // same for `y` and `#y`:

    // Pretend there are proxy traps in the handler:
    const proxy = new Proxy(this, {}, [x]);

    // This fires all proxy traps like any other property.
    proxy.#x = proxy.#x + 1;
    proxy[x] = proxy[x] + 1;
    Object.getOwnPropertyDescriptor(proxy, x);

    // Non-whitelisted private symbols do not trap.
    proxy.#y = proxy.#y + 1;
    proxy[y] = proxy[y] + 1;
    Object.getOwnPropertyDescriptor(proxy, y);
  }
}

How would you build a membrane library?

(which could work in that JS if all the primordials were frozen before the membrane library got to load?)

For that, see my Membrane implementation. Using private symbols, transparent proxies, and proxy whitelist, I was able to build a membrane that can operate even if the realms are frozen.

Using the test suite, we can assert that this design works for:

• Closure based access
  • string fields ((o) => o.field, and (o, v) => o.field = v)
  • public symbol fields ((o) => o[sym], and (o, v) => o[sym] = v)
  • private symbol fields ((o) => o.#field, and (o, v) => o.#field = v)
  • (note, whitelist API is unnecessary for closure based access, meaning it's unnecessary for syntax-only private symbols)
• Reified key based access
  • string fields (o["field"], and o["field"] = v)
  • public symbol fields (o[sym], and o[sym] = v)
  • late-known private symbol fields (o[privSym], and o[privSym] = v)
  • (meaning the private symbol was added to the whitelist after being set on a base object)
  • pre-known private symbol fields (o[privSym], and o[privSym] = v)
  • (meaning the private symbol was contained in the whitelist before being set on a base object)

Further, we can prove it works for both shadow/real target membranes, and just real target membranes.

[erights]

and just real target membranes.

What is that?

[erights]

Leaving aside decorators and looking only at the current stage 3 proposal, is there anything in your proposal that is incompatible with it? With both name reification and private symbols postponed, is what remains equivalent to the current stage 3 proposal?

[erights]

Btw, we need to ask the same question for PNLSOWNSF. I think the answer is "Yes, PNLSOWNSF is compatible with the current stage 3 proposal." But it needs more examination. I am far from confident, especially because PNLSOWNSF is, so far, only a vague direction rather than a worked out proposal.