michaelballantyne
commented
7 months ago The problem occurs because of an interaction between the way that the tests are written and the fact that we made the nonterminal syntax property non-preserved.
When applying a syntax property with syntax-property, there's an optional third argument that indicates whether the property should be preserved in serialized syntax, i.e. when a compiled file is written out by raco make or racket -y.
If a syntax property is really only needed temporarily for compilation, making it non-preserved can avoid increasing the size of compiled files. So as of right now we made the nonterminal property non-preserved.
In the tests as written, phase0-expand-flow expands the initial syntax at expansion time of the test module, but the remainder of the test (eg. calling deforest-pass and deforested?) happens at run-time. That is, this test:
(deforested? (syntax->datum
(deforest-pass
(phase0-expand-flow
#'(>< (~>> (filter odd?) (map sqr)))))))Expands to something like:
(deforested? (syntax->datum
(deforest-pass
#'(amp
(thread
(#%blanket-template
((#%host-expression filter)
(#%host-expression odd?)
__))
(#%blanket-template
((#%host-expression map)
(#%host-expression sqr)
__)))))))and then that runs at runtime. The problem is that because the nonterminal property is not preserved, when running with racket -y the property is lost after expansion time and is thus missing when running deforest-pass at runtime.
I suggest instead running the entire compiler pipeline at expansion time, and only returning a boolean to runtime. This makes sure that the compiler runs in the same way it will in a real program---at phase 1, and without any step of serialization between expansion and compilation. Here's a test that works file with both racket and racket -y:
#lang racket/base
(require
(for-syntax racket/base racket/string)
qi/flow/core/compiler
qi/flow/core/deforest
;; necessary to recognize and expand core forms correctly
qi/flow/extended/expander
;; necessary to correctly expand the right-threading form
qi/flow/extended/forms
(submod qi/flow/extended/expander invoke)
syntax/macro-testing
rackunit)
(begin-for-syntax
(define (deforested? exp)
(string-contains? (format "~a" exp) "cstream")))
(check-true
(phase1-eval
(deforested? (syntax->datum
(deforest-pass
(expand-flow
#'(>< (~>> (filter odd?) (map sqr)))))))))
countvajhula
benknoble
Summary of Changes
This began as an investigation of the
long-functional-pipelinebenchmark, which it turned out wasn't getting deforested. That seems to have something to do with the fact that that benchmark is rewritten by both of our current optimization passes, normalization and deforestation, and most likely, the latter doesn't see thenonterminalsyntax property ~for reasons that are somewhat mysterious~ because we weren't explicitly propagating it, and the syntax property being "nonpreserved" meant that our tests were not correctly handling this property across phases (see comments below).In the course of investigating I ended up refactoring a lot of tests, and they actually revealed a lot of other smaller issues that are mostly fixed. ~But the main nonterminal property issue remains a mystery~. Summary below:
racket, but fail when usingracket -y. In the latter case, thenonterminalproperty is not present after expansion.compiler/rules.rktthat passes if we manually introduce thenonterminalproperty between passes viatag-form-syntaxcompiler.rktmodule itself, at the conclusion ofnormalize-pass, does not make a difference to the test result. Adding the property at thefind-and-map/qilevel (after excluding non-syntax and expressions tagged#%host-expression-- which it didn't like) doesn't make a difference to the test result either.Other changes:
nonterminalpropertyutilmodule to (compiler)pass. There are too many modules named "util" and obviously we should avoid such a generic name.Public Domain Dedication
(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)