Make an RFC about the top level

[LiberalArtist]

We all know that "the top level is hopeless," but maybe Racket2 could be a chance to make it a little less hopeless.

The most comprehensive list of top-level issues I know is this one, but I know there have also been more mailing-list discussions in the years since. I particularly remember this thread that started with @lexi-lambda's ideas in the context of the Hackett REPL for building on submodule semantics and this one started by Christopher Lemmer Webber on their wish for a more mutable top level.

[LiberalArtist]

A mailing list discussion just drew my attention to this particular infelicity:

> (if #f unbound-identifier 1)
1
> (module example racket/base
    (if #f unbound-identifier 1))
unbound-identifier: unbound identifier in: unbound-identifier

I hadn't realized before that such references to undefined top-level identifiers are allowed even without a lambda or similar delaying evaluation, just by happening to be in a branch of a conditional that isn't taken.

[rocketnia]

I'm surprised to see that nearly all the "top level is hopeless" links involve the same situation: Someone tries to bind a top-level variable to a recursive function, and the recursive call is being treated not as a reference to that top-level variable but instead as a reference to an existing module binding or an existing top-level syntax binding with the same name.

It seems like those situations aren't hopeless for the users, because the users can usually apply unintrusive solutions: Put the definition in a module rather than a REPL/eval command; pick a name that doesn't collide with an existing name; forward-declare the top level variable and then set! it to have the desired recursive behavior; or when applicable, use namespace-require/copy so that the name collided with already refers to a top-level binding rather than a module binding.

I also doubt this is hopeless for designers of define either. I bet a variant of (define x ...) could bind x in the right-hand side, perhaps by expanding to something shaped like (define x (let-syntax ([x ...]) ...)). (Maybe it could even be defined in a library right now.)

---

In the bigger picture, there are deeper eval paradoxes that come to mind: Should the user be able to eval code that uses a macro before they eval the macro's definition? Should the user be able to eval a redefinition of a macro they've already defined? If someone redefines a macro, should all the previous eval calls that used that macro be re-expanded? If the macro has side effects, should they run again? Should the old side effects be undone? Should macros have side effects at all?

Racket's design encodes particular choices here. To the extent the top level may be "hopeless" in Racket, I think it's a result of these choices. Some languages go so far as to not to have eval at all, which is basically enforcing the "top level is hopeless" sentiment; not only are people told it's not a good idea, but the language doesn't even provide it. Similarly, a lot of languages have eval but no macros.

One of the options I find the most compelling is to have eval and macros but no redefinitions. Redefinitions are an inherent source of surprises or "hopelessness"; a user might expect the new definition to be in effect where actually the old definition is still in effect, or vice versa. As long as users understand what to expect, it's no problem, but many legitimate kinds of redefinition require nuance in the way things are defined:

• Define this beginning at the instant it's computed during the program's execution, and have it last until a similar definition arises.
• Define this beginning at the instant the program is run and ending three seconds later, and have it automatically take precedence over any similar definition that arises within that window of time.
• Define this beginning at the instant the program is run and ending five seconds later, and let it be overridden by any similar definition that arises within that window of time.
• Define this for the purposes of API versions beginning with 3.0.1 and ending with 3.0.8.
• Define this for as long as this card remains in play and attached to the target, overriding any conflicting definitions the target is making.
• Define this as an author-recommended default that can be overridden by user configuration.

One-size-fits-all definition forms like define and define-syntax aren't enough to capture that nuance. If they support any particular notion of redefinition, sometimes it's going to be an unexpected one.

[SimonLSchlee]

Languages like Mozart/Oz (like described and used in the book "Concepts, Techniques, and Models of Computer Programming) have a different model for variables and definitions. They use "Single-Assignment Variables" their semantics work like this:

1. you can define a variable (via specifying its identifier in different syntactic constructs like let, within a module, a function body etc.)
2. at some later time the variable can be assigned once which makes it change from undefined to the assigned value

What makes these semantics really interesting is that they allow you to work with partial data-structures and synchronizing multiple threads producing streams lazily etc. right from the core set of kernel operations of the language. The upside is you can easily create code where values are computed lazily or in a different logical thread, the downside is that if you forget or your code fails to provide a value for a variable, that causes the thread that needs that variable to become blocked. But that downside is not necessarily too bad, you can create tools that allow you to inspect and monitor what happens and you also could write code that causes a timeout after a while triggering an error or fallback. The big upside with that kind of semantics is that you can write a lot of programs which are declarative and execute deterministically. And there is a clear line between deterministic and non-deterministic programs.

---

Personally I think text source-code and repl's are a nice feature, but they do not feel modern and state of the art to me. When looking at things like Bret Victors code visualizations, it seems like it is time that a language integrates the tools that are used to edit, debug and run programs in that language. For me that would mean having an interactive environment that replaces the repl. You would have an interactive editor for your module that visualizes the parts that are inferrable or evaluate-able via background expansion. Instead of repl interaction you could have interactive annotation of functions with example invocations, showing you the resulting value. Allowing you to upgrade that to an assertion/unit-test.

There seem to be a lot of people who love text based source code, but IMHO I think the reason for that is, that so far there haven't been a lot of obviously well implemented "structured" formats for source code. Those formats would require more tooling and integration, especially in respect to version control of source code, but they would also elevate programming from sitting down at a typewriter. I also think that parsing and syntax could become more of a non-issue / view-your-code-like-you-want. Sadly so far it seems that all experiments in that direction haven't made the jump from academia or private industry out into the "open". Well one positive example seems to be https://pharo.org/ for a modern smalltalk, I don't have experience with it, but it seems they have both an interactive environment and version control via git. Relevant reference to comment in another issue: https://github.com/racket/racket2-rfcs/issues/96#issuecomment-517796591

7 minutes of Pharo Smalltalk for Rubyists: https://www.youtube.com/watch?v=HOuZyOKa91o (maybe racket could steal some features)