Fix weird syntax pair bug

[countvajhula]

Summary of Changes

This fixes the weird syntax pair bug, ~but we don't really understand why it happens this way~.

The reason it was happening is that syntax-e is apparently not guaranteed to produce a list even if the input happens to be a syntax list. Since Qi has some core syntax that can be used in identifier form, that meant that expansion was producing a syntax pair here rather than a syntax list (for reasons that should be considered internal implementation details that we should not rely on due to syntax-e making no promises regarding producing a syntax list).

So instead of doing a naive tree traversal where we attempt to optimize every syntax component, we had a couple of more robust options to consider:

1. Do a proper tree traversal that is aware of Qi core language syntax and only attempts to apply the transforming function to well-formed uses (rather than partial subsets) of this syntax.
2. Add a generic utilility in Syntax Spec that would provide such a traversal to any language by inferring it from the core language grammar

We went with a third, short-term option, for Syntax Spec to attach a syntax property indicating that syntax is or isn't a full and well-formed use of core language syntax. And then in our naive traversal, we look for this property and only attempt to optimize if it's present.

(this is WIP from today's meeting)

Public Domain Dedication

• [x] In contributing, I relinquish any copyright claims on my contribution and freely release it into the public domain in the simple hope that it will provide value.

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

[benknoble]

Does using #lang debug racket and its #R prefixes work to help show input/outputs?

[countvajhula]

@benknoble that's a great idea, will keep that in mind for next time! We ended up going with a more robust fix than the syntax->list by getting some insider info from Syntax Spec about which subtrees are legitimate uses of the core language that it would be valid to attempt to optimize (Michael joined today and added + released this new feature 😸. ).

Most of the compiler tests now fail since the actual logic depends on the presence of this new syntax property (from Syntax Spec) which the tests don't yet know about. I attempted to add that in the tests earlier today and most of them were passing but a few weren't, and I haven't had a chance to debug those failures yet. So, I just committed the fix we wrote during the meeting for now.

[countvajhula]

I figured I'd just commit my first attempt at fixing the tests too, for what it's worth. I haven't had a chance to investigate why these two tests are failing:

Compiler utilities tests > find-and-map/qi > does not explore node on false return value
FAILURE
name:       check-equal?
location:   util.rkt:59:4
actual:     '(a q (stop c q))
expected:   '(a q (stop c b))
--------------------
--------------------
Compiler utilities tests > find-and-map/qi > matches at multiple levels
FAILURE
name:       check-equal?
location:   util.rkt:87:4
actual:     '(a c (a d e))
expected:   '(c (a d e))
--------------------
12 success(es) 2 failure(s) 0 error(s) 14 test(s) run

[countvajhula]

@benknoble #lang debug's (debug-repl) helped identify the cause -- thanks for the reminder to use this great tool. It's now fixed but I still need to incorporate the new syntax property in the remaining failing compiler tests.

[countvajhula]

This fix looks like it's working. However, there's another issue now where the long-functional-pipeline benchmark isn't being deforested, and as far as I can tell the new nonterminal syntax property may need to be propagated between passes to solve that. My initial attempts to do that didn't work so I'll need to dig deeper (e.g. "#%host-expression: can only resume a host expansion suspension value"). But for now, I'll go ahead and merge this so that we can continue in a fresh PR to keep the scope bounded.