007 and P6

[raiph]

Would you please comment here on whether you feel happy about me mentioning 007 in relation to P6, or to Rakudo, on SO and/or other places, in the sort of manner I quote below.

In addition, please comment on the is parsed stuff I sketch out at the end of this issue post.

I'm writing this as an issue rather than a PM because I'm wanting to read a kind of "updated official public" position on the relationship between 007 and P6 that's integrated with your latest personal feelings on the topic.

---

In this SO question JJMerelo sketched the following approach to a problem he had:

sub postfix:<.&>( $sub, **@args) { ... }

His code was far, far adrift from reality for multiple reasons but I commented that:

Aiui something like it may one day be possible if 007 evolves to cover this case and gets merged back into a future P6.

I didn't necessarily mean the specifics of a postfix routine, or the signature he had, but rather the general idea of one day having a fair degree of freedom to get down and dirty mid parse with macros.

Also, I could have written:

Aiui something like it may one day be possible if 007 evolves to cover this case and gets merged back into a future Rakudo.

(i.e. substituting Rakudo for P6)

---

On reflection I wonder if 007 is loosely analogous to Parrot as it was around 2006-2009, in the sense that it's its own vibrant thing and its future relevance to P6 and/or Rakudo is on a trajectory to be more an inspiration for a very loosely similar thing (cf NQP/MoarVM) rather than as a thing that "gets merged back in".

If you're very unsure about such matters, and if that's not so much because it's hard to figure out what I'm getting at, or how to talk about it, but more that it's too early to know so might go either way, would you say that's mostly about the need to keep it fun -- cf Audrey's -Ofun -- or more that the technical challenge is driving you away from the specific nature of P6 and/or Rakudo as it is today?

I'm rooting for you to keep pushing 007 forward regardless of P6 or Rakudo. Aiui, figuring out a way to get lisp macro like features working in a non-homoiconic language is 007's purpose and that seems a worthwhile goal in its own right, of far more global interest than anything specific to P6. (And of course generating global interest is quite plausibly the only way to really work in P6's interest. Again, cf Audrey's -Ofun.)

In the meantime, I want Rakudo and P6 to push forward regardless of 007 and to one day get macros that cover cases such as JJMerelo's.

And I'm trying to figure out how that's going to look in the 6.d and 6.e I project in my imagination.

---

To be highly specific again, I'm wondering if there might be an is parsed or similar in which it's possible to have the compiler, when parsing code like this:

foo.&bar: baz

allow a macro to do something like the following:

• Compete to parse the . and following code (currently the compiler gets to it first and, aiui, wouldn't give a macro call a chance);

• Call back into the usual grammar/actions mechanism to attempt to resolve &bar as the normal grammar/actions would normally do, fail to find it (per JJMerelo's case, &bar doesn't exist), then get control back to continue in the macro in such a way that it's realistically viable to continue with the parse to a successful outcome;

• Call back into the usual grammar/actions to parse the : and baz as they would normally be parsed but again get control back so the AST could be massaged as desired.

---

Perhaps this should have been multiple issues or no issues at all but rather some other form of communication. Please redirect as you think appropriate.

Last but not least, thank you for 007, wherever you take it.

[vendethiel]

To be highly specific again, I'm wondering if there might be an is parsed or similar in which it's possible to have the compiler, when parsing code like this:

seems like a is parsed macro would want to be able to do this, yes:

sub postfix:<.&>(a) is parsed /<call>/ {}

[raiph]

Thanks for responding.

Hmm. The immediate rubber duck effect on me is the intuitive sense that slangs are more appropriate for this.

Since I opened this issue I've encountered another SO question that's in slang-or-macro territory and I've again included a reference to 007 in my answer to it.

This latest example -- turning auto $s = 'foo'; into my Str $s = 'foo; -- directly suggests a statement macro to me, not a slang.

[masak]

Just saw this. First off, regarding the very last paragraph above; yes, it suggests a statement macro to me too.

Longer reply coming to the OP.

[masak]

Addressing the immediate question first:

Would you please comment here on whether you feel happy about me mentioning 007 in relation to P6, or to Rakudo, on SO and/or other places, in the sort of manner I quote below.

In a word, yes.

I think the wording you're asking me whether I'm happy about is this one:

[...] if 007 evolves to cover this case and gets merged back into a future P6.

Yes, that's the primary goal of 007, and always was.

The word "merge" might be overstating how simple the transition back into Perl 6 will be, of course. It'll be a merge of ideas/concepts/techniques rather than of codebases. But given that that is understood, I have absolutely no problem with the way 007 is mentioned in relation to Perl 6.

[masak]

Immediate afterthought: I'm thinking I should finally replace the README.md file with something a bit more... official, which both clearly shows what 007 is, and what its goals and priorities are.

I will open up a separate issue for replacing the README.md file.

[masak]

@raiph — is this issue closable, with the above answers?

In the meantime, I figure I'll keep going and try to sketch out a sort of "current status" of 007. Some of that will likely find its way back into the ROADMAP.md.

[masak]

I just went back and checked some of the earliest versions of the README.md, before it got its current license to kill. Here's the initial description of 007:

A small language. A test bed for macro ideas.

(I also got nostalgia for how simple and malleable things were back then. Oh well.)

[masak]

The next clue is in the middle of the Bond cold open:

"You will never achieve the simplicity of Lisp macros in that way," says the contact, as he slowly gets up.

"True, but we can get quite close. And the aim is not to emulate Lisp slavishly, the aim is to produce a Perlish macro system that can serve Perl users as well as possible," explains Bond.

(I still like the current README.md. I think it explains a lot. But I also respect those who think it's a struggle to slog through a sort of allegory-baked-into-a-pastiche.)

[masak]

If you asked me today, I would say that 007 is a test bed for giving Rakudo real useful macros. (Whether I'd say "Rakudo" or "Perl 6" doesn't really matter to me. Currently Rakudo is the only viable implementation of Perl 6.)

007 keeps toying around with a number of other goals, too. Copying from the ROADMAP.md file:

Work on 007 falls into two main tracks:

• Features that help explore macro-like things (ultimately for Perl 6)
• Features for 007 the language (ultimately for 007)

The first track is still the raison d'être for 007. The second track rounds 007 off as a nicer tool to work with.

Examples of the latter include: giving the language a well-rounded set of built-ins, making the runtime and the parser bootstrapped, achieving some sort of minimalism in the core concepts of the language.

The biggest advance in understanding what quasis and macros are, which will ultimately help Perl 6 a lot, is documented in this issue, but I can summarize it here as quasis are not represented in the AST as Qtree fragments, because they are not fully formed yet; instead they are represented as (template) functions that take the evaluated unquote expressions as parameters and return a Qtree fragment. The realization that this is how it has to be sprung out of the fortuitous confluence between some unrelated consulting work I did last summer, and a (since-abandoned) foray I did into giving the whole Q hierarchy declared types.

I doubt I would have had the same realization that soon, if ever, if I had been working with macros within the confines of Rakudo. And that, in the end, is the whole idea of 007: to scout out ahead, and explore what macros should be and must be, to ultimately be useful and usable for Rakudo/Perl 6.

[masak]

A few comments on the status update from December follow. This will likely be slightly incomplete, but I'm aiming for "better than nothing".

I've gone full circle, trying a very difficult thing first with strategy A, then with B, and now with A again.

Surprisingly, I/we haven't fixed #212 yet. It remains very nearly fixed, but I haven't gotten it all the way yet. I think this happy announcement was a bit premature, and I suspect what it's blocking on is a proper resolution of #212. I don't want to oversell it, but #212 will fix a lot.

Why hasn't it gotten more work so far? To be honest, I think I've been dragging my feet because whenever I start thinking about these issues, my brain melts and takes a while to re-congeal. The really useful instrumentation à la #222 would help here, but... putting that in place melts the brain just as much.

Central to this is the term "injectile". Briefly, if you do my x = 5; inside of a quasi, we already said that that statement is only properly represented as a Q::Statement::My in a template function somewhere. But for every time that macro is invoked, that declaration will end up somewhere new in the code, and the right compile-time things need to happen to make sure each (new) scope has an x variable declared. Those new scopes are our injectiles — the things that get injected into code via quasis-filled-with-unquote-parameters via macro calls.

Things that I'm looking forward to right now:

• Landing macro infix:<ff> in master, which is quite literally one small fix away at this point.

If you ask me, #212 and then #207 should still be my/our utmost priority, simply because they are well within reach, and would mean a lot if they worked.

• Landing a huge object system refactor that really cleans up the language.

The PR itself ended up diverging too much and getting closed without merge, but it ended on a very happy note.

The first priority when picking up the pieces from that PR should be the Object/Dict split (#184). We know we want to split those two apart anyway, and the sooner the better.

I'm less optimistic about doing the rest of the object refactor anytime soon. It needs to happen, but I think it kind of needs to happen on the side, prototyping a new object system from scratch and then wholesale-replacing it.

Speaking of prototyping from scratch, I filed #293 the other day, and I'm hoping we'll have something soon that shows the feasibility of dynamically adding things into grammar categories as we parse. (Including also dropping those custom categories as their declaration goes out of scope.) I've long seen it coming on the horizon that a Perl 6 grammar won't do this for us — it kind of does with operators, but that's because we try very hard and leave the comfort zone of the grammar itself. I think having more control over the parser will end up being a good thing.

Making a better web site, focused around language tutorials and API documentation.

I would still very much like to see this happen. One big step towards making it happen is that we now have in-code documentation which is meant to go out directly as documentation on the web page. Work still remains, though.

Besides these, many many smaller things would individually help clean up 007 and make it more able to do what it was meant to do. Quickly scanning through the issue queue, I see that we would be helped by fixing #10, #80, #133, #145, #173, #183, #216, #245, #247, #250, #254, #256, #257. These are somewhere between cage-cleaning and "hey, this is wrong, we should fix it before it causes some damage". (Note to self: create an issue label to mark these up somehow.)

In related news, I have thought quite a bit about modules lately. I think it would help a lot for 007 to have a module import mechanism. #53 which handles this is stuck in an exquisite indecisiveness about whether or not to forcibly inject identifiers into the importing scope; it's possible we just need to try stuff out to see how it works. (Why do I think modules and imports are important? Because I think macros are often defined in one compunit and then imported and used in another. Exploring macros then means exploring how they interact with modules, too.)

I'm going to stop here. Likely I will think of more things as soon as I press the "Comment" button. :smile:

[masak]

Another thing 007 has done, which I'm not even sure I was actually hoping for when I started out three and a half years ago, is to pin down what a macro actually is. Or, more precisely, under which circumstances one might be inclined to use a macro rather than, say, a normal subroutine or some other existing language feature.

I had some very recent insights into this. I should probably write about them, either here or as a blog post.

[masak]

What are macros? I think it's about as easy to arrive at the wrong answer as it is with "What are objects?".

Back in 2011, all I had was basically LOG. See this post, for example. You can't motivate an entire macro system with all its inherent complexity just to be able to define LOG.

Nowadays I would say that macros are for "offsetting computation". A few examples:

• Need to check whether a thing x.y().z["w"] exists in a dictionary? You could use regular methods on the dictionary, but there's a certain charm in actually writing out the path like that and do exists x.y().z["w"]. You can only do that if exists is a macro, since a regular function or operator would only get the rvalue... provided the lookup doesn't fail completely and throws an exception. The macro exploits our natural propensity to think x.y().z["w"] will work. It will, but only under the slightly offset computation that exists provides. (#292)

• Ditto delete. (#290)

• Use expressions not to evaluate them but to signify the shape of expressions, in order to "smartmatch" other expressions. (#284)

• Use expressions not to evaluate them but to find an unknown x that satisfies the expression, and makes it truthy. (#268)

• (Latecoming update:) The lvalue or location macro for getting a Location instead of an rvalue of an expression. (#214)

Note that all these examples don't change what expressions look like, only what they mean. They temporarily change (or defer, or disable) the evaluation semantics of the language. This is a force multiplier because you get all the same constructs as before, but with new meaning. It's also something that regular subs and operators definitely won't give you.

I wish I had understood this much sooner. :smile:

[masak]

In related news, I have thought quite a bit about modules lately. I think it would help a lot for 007 to have a module import mechanism. #53 which handles this is stuck in an exquisite indecisiveness about whether or not to forcibly inject identifiers into the importing scope; it's possible we just need to try stuff out to see how it works.

Happily, just writing this made me land a design for modules and imports. So now we're much further; just waiting for an implementation now.

[masak]

Let me quickly address JJMerelo's StackOverflow question. It's interesting to me mostly because it reveals something about (wrong) expectations of Perl 6, and perhaps of programming in general.

The intent JJMerelo expresses is fairly clear: he wants any kind of method to exist on his Nothing class. (Liz++ provides an excellent answer, replying to this intent and translating it back to working Perl 6.) But let's focus on/unpack why JJMerelo's approach doesn't work.

He seems to think that postfix:<.&> handles method calls. This, as far as I know, it does not do. Oh! He has gone to this documentation page and (wrongly) concluded that this is an operator that can be defined. Not only can it not be defined/overridden — it's not even an operator (as the documentation points out). Rather, it's the fact that you're allowed to put any variable such as &f after the method call dot. (This seems to be handled by token methodop in src/Perl6/Grammar.nqp.) One also needs to know that &f is the "actual name" of a sub f in the lexical environment — the shorter form f is secondary, and used as a way to call rather than refer to the sub.

Next up, it's attractive to think that declaring a sub inside a class might be the right thing to do. But I just see a sub scoped to a class block — I know that by design it will have no effect outside of the class, because a sub doesn't interact with or register with the class in any way. Subs in classes are useful, but mostly for small helper tasks unrelated to self. (I know of no other language that allows both subs and methods in the top level of a class.)

Lastly, you raiph point out that this is more the domain of macros, not subs. Let's pretend that the two show-stoppers above did not exist. Then the problem is still that macros have a very different scope/axis than classes do. In this case we happen to know that we're calling a method on Nothing, a type object, but in the general case, we're just calling a method on some variable which may contain any object which in turn may be an instance of any class. This "late binding" is one of the defining features of object orientation, and macros do not match up well with such late binding. A macro is supposed to have done its job already during the compile phase, but here we have something that's not determined until runtime.

I don't like being the bearer of bad news, but I don't hesitate to say that a macro will never do exactly what JJMerelo hoped his sub would. The closest feature request we have on file for 007 is #203, which also happens to be on the prestigious shortlist of things that will end up in the new, better examples/ directiory (#194). Note how this macro is not connected to or defined in a class; instead it operates on all left-hand terms equally.

[vendethiel]

I don't like being the bearer of bad news, but I don't hesitate to say that a macro will never do exactly what JJMerelo hoped his sub would.

why coudln't a macro generate a runtime type check for a variable? :P Seems to me like a macro could generate a if, so while not quite beautiful/pretty, it could amount to the correct behavior.

[masak]

You mean something like this?

if {{{lhs}}} ~~ Nothing {
    throw new X::NYI(...);
}

Yes, we could surely do that, but (a) whereas the Nothing class only needed to answer what happens to method calls to (instances of) itself, a macro containing the above would need to have a story for all method calls — that was my point above, that objects sort of cut down whereas macros cut across — but fine, we could imagine that the above if has an else where we somehow delegate to the normal method behavior; nevertheless (b) we'd still be solving something with a macro that's better solved with MOP or something (such as the FALLBACK solution in the SO answer). The above is going against the grain of macros.

Just as it's important to know what macros are and how best to use them, it's also important to know what they are not (at least not easily), and when best to use something else.

[masak]

Oh, and we'd also need a mechanism for overriding an existing grammar rule while also being able to default back to it in some cases. I still don't know if that's an excellent idea; at best, I think we'd run into problems where such overrides don't compose very well.

[raiph]

Thanks for your thoughtful responses. You can pretty much assume I was thinking positive stuff like "yes" or "awesome" to everything you've written and all questions you asked so far unless I comment on it below.

quasis are ... (template) functions that take the evaluated unquote expressions as parameters and return a Qtree fragment ... Surprisingly, I/we haven't fixed #212 yet ... whenever I start thinking about these issues, my brain melts and takes a while to re-congeal.

These comments and some others triggered a brainstorm. In retrospect it's probably an idea more suited to being directly hacked into an 006½ branch of Rakudo rather than being of relevance to 007.

I don't like being the bearer of bad news, but I don't hesitate to say that a macro will never do exactly what JJMerelo hoped his sub would.

Right. I was thinking that a macro might be able to do better than a slang tweak for competing with the compiler to get at the parsing of a .. But as I hinted after vendethiel's reply about is parsed I've shifted to thinking tweaking via a slang would be better than a macro for getting at the . parse and doing the sort of weaving in and out of the existing parsing that would be necessary to do the sort of thing JJ was talking about.

[masak]

Since this issue was resolved after replying to the "mentioning 007 in relation to P6" and is parsed questions in the OP, I've closed it. But I would like to write a few words about my personal goal with 007, or rather 007's goal as a language.

In October 2014, I implemented a super-tiny Lisp called ipso. The paper The Roots of Lisp was my inspiration, and I realized it was a really powerful thing for a language to be able to implement its own evaluation semantics in half a page of code, like on page 8 of that paper.

Even though the implementation worked fine, I was left a bit disappointed.

I feel like there's something awesome I'm not quite getting about the metacircularity I just implemented. — masak, 2014-11-02

The presence of a metacircular evaluator didn't lead to a miraculous increase in programming power. It was literally just a tiny language whose in-language eval worked exactly like its meta-language eval. (Except slower.)

If I worked hard inside the language to define a new, improved in-language eval, then it would suddenly not be the same as the meta-language eval any more. I failed to see the point/charm, and maybe I still do.

(By the way, ipso is not dead, just sleeping. Maybe one day I'll go back and give it lexical scopes.)

Worse, I saw that starting from a tiny metacircular Lisp and building from there would never get me to a Perl 6 with macros. (Or more exactly, I never saw how starting from a Lisp would get me to a Perl 6. Lisp is opinionated, assumes stuff. Those assumptions will not go away along the way.)

I can't find the quote for it now, but sometime after that I told [Coke] or PerlJam that what I'd really like was a bigger metacircular loop, where the language explained how to parse itself, and then to run the result. The reply I got was "sounds not so minimal anymore".

Shortly after that (or shortly before, I don't remember), 007 was born. Its distant goal has always been to "close the loop", so to speak, and be able to express a Lisp-like metacircular parser that (a) is as capable as Perl 6's, or can be extended to be as capable, and (b) is "minimal" in the sense that each of its parts is there in service of something the parser itself needs.

Part of this work consists of figuring out a kind of "dependency ordering" between all of the language's features. At its core are things that we've taken to be axiomatic. Lambda calculus kind of shows that you really only need to do that with functions; 007 will have a slightly bigger core. My current rough list includes the primitive types, arrays, dicts, objects, functions, various control flow.

Lately I've realized that it's better if compile to bytecode, not just to an AST. It's far too easy to accidentally cheat and rely on some AST information that shouldn't be readily available at runtime. (The classical example of where we do that right now is with routines, which expose all of their Qtrees when they should really be opaque at runtime; see #253.) The bytecode represents a machine-independent semantics. I'd say the language is still metacircular if a bytecode evaluator can also be implemented in 007.

Another reason for minimalism and a small core is that if something could be defined outside of the core, possibly as a macro, then it should be. If nothing else, this activity of defining complex things in terms of simple/core things will help exercise the macro system in various ways.

If I'm thinking about this right, even if statements should be definable rather than axiomatic under this scheme. The parser would just generate some computed goto-like instructions in the target bytecode. The evaluator would just evaluate those instructions as a 007 if. That's metacircularity for ya. :smile:

(Note: if is central enough to programming that I'd still want it as core, so no import would be needed to use them.)

Anyway, those are some unsorted thoughts about what propels me forward, and still (uncharacteristically) keeps me interested 3.5 years later. I want to build 007 as a language that explores not just what it means to be an Algoloid with macros, but also what it means to be an Algoloid that uses macros to define itself.

Somewhere in the IRC backlog I found a quote from myself saying that being a language designer means entertaining several conflicting views at once in order to compare them. As I thought about writing this comment I realized that in some ways with 007's issue queue, I've recreated the heydays of p6l's language discussions, so I could relive them with my own language.

I definitely want a book to come out of this as the end result.

[raiph]

what it means to be an Algoloid that uses macros to define itself

Is Nim such a beast?

[masak]

I was aware of Nim, but not deeply. A quick scan of the Wikipedia page tells me that it's bootstrapped and the compiler is written in Nim itself. So... probably?

Adding Nim to the list of languages to look at. Syntactically and semantically it looks different enough from 007 that I don't feel it's just a "better 007". But my bet is that there are still things to learn from Nim, as it does look similar.

[masak]

I suppose the attractive thing about programming languages being defined/definable in themselves is that it hints at a certain "completeness", but also a very democratic idea of "core is userland" and of dogfooding.

It's also totally what mathematics tried to do with formal systems and what Gödel showed to be impossible. 😋