Syntax-based PTC makes adoption/migration harder

[getify]

I have mentioned this in other threads, but it's important enough that I don't want to have it get lost in the shuffle.

I understand why explicit syntax is being asked for, but one major downside of an explicit syntax for opting into PTC (beyond just literally making a PTC-formatted call) is that it makes adoption of recursion patterns harder than the implicit PTC did.

Background (Implicit PTC such as it is)

Right now, sans PTC, if you want to author code using recursive algorithms, you have to be aware of the potential stack limitations and avoid altogether any use of that code that could go out of bounds. I posit that this reality has, for the better part of 2 decades, acted as a depressive effect in the potential adoption of recursion based programming (such as that which is common in FP). So, one reason why we don't have sets of recursive code to justify the efforts of PTC implementation is because the places where you can use recursion have been artificially so limited that the effort hasn't been much worth it.

Now, introduce implicit PTC with ES2015. The game starts to change. The message becomes: "eventually, you'll be able to do all that fancy recursion stuff right here in your favorite JS apps!"

So what do we do with "eventually"? Some (probably the vast majority) said, "yeah, yeah, we'll think about this once it's available, so it'll be years before it matters." This is a harsh cliff sort of approach. We can't even consider adopting (and teaching about) general recursion patterns until after all those old non-PTC browsers have died away. Especially when we think about mobile, that's not months, that's literally years or more.

Others, like myself, began looking for a bridge strategy to migrate/adopt these patterns progressively. I came up with a sort of hackish adaptive recursion pattern, that looks roughly like this:

"use strict";

function foo(x) {
    function _foo() {
        if (x > 1) {
            acc = acc + (x / 2);
            x = x - 1;
            return _foo();
        }
    }

    var acc = 1;

    while (x > 1) {
        try {
            _foo();
        }
        catch (err) { }
    }

    return acc;
}

foo( 123456 );          // 3810376848.5

Before we bikeshed on this code, I'm not claiming it's pretty or ideal. There are many potential downsides to it. It is not intended to be permanent code, but it is intended to survive the gap from non-PTC engines to PTC engines.

What I'm claiming is that it's an interim step that aids in migration to and adoption of recursive patterns. The thinking is, that implicit PTC return _foo() call is going to eventually progressively enhance to being real unbounded PTC-driven recursion. But in the meantime, the hacky and less efficient fallback is that the try..catch catches and suppresses when a browser kills the call stack, and then the while loop resumes.

I describe this pattern as a kind of meta-programming for adaptive recursion. The advantage is that this code can be written and forgotten about. You can come back and make it nicer later once the all-PTC reality arrives, but you don't have to. Over time it will perform better -- assuming of course browsers keep their promise to implement the standard.

Present (Proposed Explicit PTC)

If the suggestions of this proposal are adopted, essentially no PTC is likely (certainly not reliable) without code that uses explicit (and new) syntax.

That creates the same sort of downside "cliff" as I described above, which means that I cannot reasonably write a single snippet of code that progressively enhances itself to PTC as engines evolve. I can at best write two separate versions of the code, and then do a hard fork to one or the other based on some hacky feature test for supported syntax (like try { .. eval( "..") } catch (err) { .. }).

This syntax-based feature also likely wouldn't be transpilable practically, as the general form achieving the effects of PTC through code rewriting would be, I'm imagining, significantly complex.

So the much more likely scenario is like what's happening with other non-transpilable features introduced in ES2015, like Reflect, various WKSs, Proxies, etc: most people will just not learn or adopt this syntax-based PTC until some much later future when they have no non-PTC engines to contend with.

---

So the crux of my biggest objection to the explicit syntax for PTC is that it poisons the migration path sufficiently to the point where it will likely defer more widespread adoption of general recursion patterns for several years, at best.

[bterlson]

Are there any bridge/adapter patterns that don't involve relying on the non-standard/future-hostile semantics of OOM error exceptions being catchable (and also relying on the recursive function being provably exception-free)? Also curious to check out what others have done if you can link to them.

[littledan]

I responded to this in another place, but I just wanted to move that comment over here:

@getify There is no spec which says that a stack overflow will throw any particular kind of recoverable error, as opposed to crashing the page (as out-of-memory conditions do on many platforms), or allocating all of the memory in the system for the stack which could cause another OOM at any time. Therefore, in a pre-PTC world (e.g., the world of stable browsers), this pattern would not be guaranteed to work per-spec.

This pattern as written also seems dangerous for catching all errors, which could cause issues if an unexpected error is thrown. Further, because the extra driving while loop is needed, the terminating condition needs to be written in two places, the function doesn't have the flexibility to tail-call into other functions reliably (without an additional tracking variable), and acc and x need to be held in mutated variables, rather than parameters of _foo, making the pattern imperative and limited rather than functional and flexible. I don't see the advantage of this idiom over simply putting lines 6-7 in the body of the while loop, even if PTC does ship in some browsers.

[getify]

There is no spec which says that a stack overflow will throw any particular kind of recoverable erro

It sure seems like a de-facto standard from a web compat perspective. I've tested extensively. I didn't find any place where browsers just let the recursion go unbounded until crash. I'd be happy if someone could point to a device/browser combo where this doesn't hold?

All of the supported envs where my code needed to run, the error was thrown for restraining the call stack. Pretty sure it was always a RangeError as well.

This pattern as written also seems dangerous for catching all errors

I don't make it a habit of letting my errors in recursive routines go unhandled -- I handle them proactively. So any error that propagated out to that try..catch would necessarily be something where the engine was throwing it. It's also possible to check to make sure the error you catch to resume is actually a RangeError, for example.

But anyway, this is just a quick sketch of the pattern. That's not literally the code I put into production.

[littledan]

@getify Could you point to an example of the kind of code you put in production?

[bterlson]

I do not believe this is a pattern that gets used widely and I think there are major drawbacks that make such a pattern inadvisable for developers to use. As such I don't believe this should be a major constraint in our decision making process.

Regarding handling of OOM errors and standards, it's likely true that it is non-standard web compatible today (although I have heard in the past that some impls may not allow catching OOM in all cases) but it is rare in production code to depend on catching these errors. It's an area I am hoping we can clean up the spec more in the future. I would not advise anyone take a dependency on non-standard semantics regardless of web compatibility. Worst case, it constrains further tightening if we want to standardize this. Less worse case, but bad for developers using it, we have to evangelize the very few sites that depend on these semantics to change their code.

[getify]

I appreciate your spec-centric view of this, but my perspective is non-spec-centric, and rather more pragmatic. What's frustrating is that you discount this idea as "not widespread" and yet really offer no other suggestion for how developers can get over the cliff between "non tail call" and "tail call" other than just waiting.

I never claimed this pattern was ideal -- far from it -- but I do think it's reasonable to bridge the gap from where we are to where we're trying to get in a more graceful and progressive way than just continuing to wait.

The fact is, moving tail calls to explicit syntax makes that cliff harsher. Disregarding care for that concern means that adoption/migration to tail calls doesn't register very highly on your priority list, which is where my frustration is coming from. In general, I wish spec bodies evidenced more care for these adoption/migration paths rather than inventing features in future bubbles like "once it's already there, everything is great".

[bterlson]

Please don't confuse my position in this case with a general statement about adoption/migration paths, which is something I (and others involved in the standards process) consider extremely important. Also your point about impact on our priority list is not true (and irrelevant anyway if it's not a pattern people use in practice). My position is that for tail calls there is not a reasonable bridge that I would feel comfortable telling developers to use. I am willing to change my mind, however, if you can provide the requested data on usage and work others have done.

[getify]

usage and work others have done.

I don't have any of that. I only have what I've presented, and the one production (unfortunately non-OSS) code base that I put it into. Well, that and the fact that it's included in my ES6 book which has a fairly strong readership.

---

I would remind you of a conversation we had in person about a year ago, which I think is still the case: that it's a mistake -- I think -- to rely so heavily on real-world usage patterns (around attempts to get tail calls patterns like recursion in code) for judgements about relative importance of the pattern itself, or how the pattern will/should work.

This is a classic case of chicken-and-the-egg. It'd be like saying "no one is really using the Proxy pattern, so it's probably not all that important." No one can use the Proxy pattern until it lands.

There is not widespread usage of this pattern (or anything like it), but that's not just because it's unimportant or unattractive, but because we haven't been able to (until ES6) ever have any hope that recursion could be used in general/unbounded cases. There are a number of communities, like the FPrs, ClojureScript, etc, that have long expressed the desire for stuff like tail calls, but they haven't been able to use any code remotely like it because JS didn't have tail calls.

Even as soon as ES6 landed, there were rumors that no browsers would implement it "any time soon". Of course devs in those communities haven't been early movers trying to get toward those patterns. They have plenty of other ways to do it, like trampolines, etc. Inertia is hard to overcome, especially if you're doubting that the "better" support is coming any time soon. That doesn't mean they don't want to, or it's not sufficiently important or desired, it just means they haven't.

What I'm arguing is that this pattern (or something else, better?) for migration should be more important. It should be a path, even if less than ideal, where any of the communities (besides my own) try to move towards using tail calls.

They definitely won't if concrete syntax ends up being required. They'll wait until all the non-STC engines are long gone. When you count mobile update cycles, that could be 3-5 years or more.

What I wish is that we could have a verified and reliable path toward migration that we can do now instead of just waiting. I wouldn't leave that code in once non-PTC engines become irrelevant. But I'd rather write with those tools now, for the next 3-5 years, rather than just keep waiting. And waiting.

[bterlson]

You seem to be misinterpreting my argument, so to restate more simply: there is no known adapter pattern that I would want to encourage people to use, and further, regardless of it I would encourage it, I don't see what you have presented getting used. If there were a reasonable bridging strategy (or those that I consider unreasonable were nonetheless getting used) then I would reconsider this position.

[getify]

And my point, restated simply: no bridging pattern has any chance at all of existing or being adopted if we put a hard syntax gate on this feature.

[bterlson]

Agreed, and since there is no bridging pattern for PTC either (other than transpilation, I suppose), it's not a point for or against either proposal.