Basic explanation

[DalekBaldwin]

I wrote this a little while ago: https://news.ycombinator.com/item?id=9280737 and I thought there had to be some other way to allow two-way information flow between nodes in the syntax tree without using a code walker.

This is basically an extension of a trick I've used a few times before: you can put any data in a symbol-macrolet or macrolet binding, not just code that you actually intend to expand somewhere. So a handler can set up a context by stashing some information in a macro binding, and other macros further down the tree can inspect that binding and "signal" a condition by destructively modifying the data stored there. This doesn't halt the current expansion; the inner macros need to return some code that is at least valid enough not to trigger a genuine compile-time error.

An expansion-time analogue of handler-case can attempt to expand its body inside an flet binding. Once the region of code where the handler is active is fully expanded, the outer macro that established the handler inspects the environment again to see what was signalled, and either expands into a call to the function defined in flet if everything was okay, or invokes the actual Lisp-level condition-handling code to attempt a new expansion with whatever additional changes are needed to fix the problem further down the tree. If a condition was signalled within that expansion, no references to the function defined by that expansion are ever inserted in the code, so a compiler should be able to just strip out all the overhead for everything except for the final, completed expansion.

I think you could implement analogues of all the major features of the CL condition system this way, or you could create a condition system with different semantics. I'm not sure how much of the standard CL condition system is useful in this context. You probably don't want to invoke interactive restarts at compile-time, and expansion-handler-bind wouldn't really accomplish anything you couldn't already do with a simple macrolet. expansion-catch wouldn't allow for a notion of expanding code differently depending on whether an expansion-throw occurred, so it couldn't really react to specific conditions occurring further down the code tree. In runtime code, it can be perfectly okay for the program to keep chugging along even if a sub-computation couldn't finish because of an exceptional situation. When you're writing a set of macros that make up a DSL, that's never acceptable. You need to generate working, reliable code based only on foreseeable, non-erroneous conditions. There may be a way to use this trick to enable a more general notion of compile-time continuations, but I'm not sure how useful that would be without seeing more potential examples of DSLs that could benefit from such a system.

[DalekBaldwin]

@guicho271828

[DalekBaldwin]

@malisper

[DalekBaldwin]

(this is the simplest way to "group message" people on Github -- I just thought you guys would be interested)

[malisper]

@DalekBaldwin I just started using cl-walker to implement a version of iterate and as part of it, I came up with a trick for allowing two-way information flow. You can read about it here. Although it currently only supports two passes, one in which information is pooled together and a second in which that information is given to the individual forms, it should be possible to build a more general version of the technique I used.

[DalekBaldwin]

I don't know enough about the existing iterate package to be able to tell just how different your approach is. The original one has its own private code walker but it's tangled up with domain-specific code.

But I see that yours can at least handle things like:

(let ((stuff nil))
  (iter 
    (macrolet ((my-for (&rest args) `(for ,@args)))
      (my-for i from 1 to 3))
    (push i stuff))
  stuff)

;; (3 2 1)

so it appears to solve at least one of the problems Masataro pointed out.