Closed bitwalker closed 3 months ago
BTW, I was going over the docs, and I think after this PR this is no longer true, and should be udpated:
Procedures invoked via the exec instruction, are inlined at their call sites during compilation. Thus, from the standpoint of the final program, executing procedures this way is indistinguishable from manually including procedure code in place of the exec instruction. This also means that procedures invoked via the exec instruction are executed in the same context as the caller.
FYI, now that #1287 is merged, I need to rebase on those changes. I'll get to that tomorrow probably, but until then it's going to show like there are a ton of conflicts, just ignore that for now.
I've rebased this on next
, so it should be mergeable again shortly!
This is ready now and has been rebased on latest next
, all review comments have been addressed, aside from the removal of the Assembler::*_in_context
methods and the AssemblyContext
, as that is a larger change that feels like it belongs in a separate PR once this is merged.
A number of improvements and other features/changes have been discussed here, but I believe everything we intended to tackle as part of this PR is more or less complete.
This PR is a major refactor of the assembler crate, with various bits of new functionality in the
miden-core
andmiden-assembly
crates. There are some changes made in other crates, but these are largely tests, where the interaction with the assembler has either changed in some way, or the output being tested has changed. The following section lists the most significant changes to be aware of - keep in mind, that as I laid out in my note at the top, the changes are described in more detail in the commit messages, so read those for more info, but I'm happy to answer any questions you may have as well.Summary
Here's a high-level overview of what is contained in this PR:
lalrpop
, and the lexer generator islogos
- both are chosen because of my prior experience in compiler projects with them. They are solid, well-built crates, with a decent community around them, and both support no-std builds.thiserror
andmiette
crates (forked versions currently, for no-std support, untilerror_in_core
is stabilized). These enable very ergonomic error types (viathiserror
and#[derive(Error)]
), and the ability to layer diagnostics on those same types (viamiette
and#[derive(Diagnostic)]
). The diagnostics infrastructure is quite configurable, and I suspect we may want to take advantage of its JSON reporting backend for running the VM in the browser, so that we can extract error information in JavaScript and render pretty errors in the browser-based editor.PrettyPrint
trait, which is implemented inmiden-core
undercore/src/prettier
, by describing how you want to lay out the components of the syntax to be rendered. The prettier algorithm then takes that description and renders it using a language-agnostic algorithm. I have implemented pretty printers for both MASM and MAST, but feel free to modify their implementations as you see fit. One of the interesting things you can do with it is describe alternate layouts, which the algorithm will choose between depending on the available width of the output window (which defaults to 80 columns). You can see this in effect withspan
in the case of MAST, which will alternatively render single- vs multi-line depending on the size of the span.Instruction
, which hasn't changed much), others are expanded with new functionality or metadata (e.g.InvocationTarget
), and others are entirely new, or rewritten so as to be essentially new. Virtually all of these nodes now implement aSpanned
trait which returns the source span for that node, or a default one if there is no source code.Assembler
internals. Some parts of this are virtually untouched (particularly the translation to MAST, with the main exception being how procedure calls are handled). The way in which the assembler is instantiated in used is slightly different, but much more powerful now. These changes are all driven off the new "module graph", which is used for global inter-procedural analysis during assembly.Changes to MASM Syntax
I'd like to devote a section just to this topic, as they are both a motivating reason for some of the changes, as well as a result of reimplementing compilation in a more principled fashion:
export."_RNvCskwGfYPst2Cb_3foo16example_function"
(an identifier generated from Rust's mangling scheme). The default "bare" identifiers still have largely the same constraints as before (start with an alphabetic character, contain only alphanumerics and underscores, etc.).export
their syscalls, andproc
their private internal helper functions. I actually can't recall now if that was the case before, but it certainly is now.exp.uXX
instruction, which currently only accepts a literal.I think that largely covers the changes to the surface syntax, at least that I can recall off the top of my head here.
Changes to MAST/Assembler
One of the major things that falls out of the
Assembler
refactoring, is that we now always visit procedures in reverse topological order, i.e. callees before callers. As a result, we always know the MAST root for every procedure being called in the program. Instead of inlining the body of every procedure at every call site, the assembler now emitsPROXY
blocks for procedures called viaexec
(andCALL
forcall
, but that's not new). This has the effect of making the emitted MAST drastically smaller. I added support in the processor for executing proxy blocks (it was an error previously) - if that is not the correct behavior, we may want to revisit this as part of the larger MAST refactoring. The only downside is that the textual MAST now hasproxy.HASH
where there used to be inlined code, but even that is something we could easily address in the formatter if we actually have a problem with this. I have added a test helper though that makes expressing tests that expect certain textual MAST output that contains calls much more readable, and less fragile to changes in the standard library, etc.The other significant change to the way the assembler works is that the procedure cache has been rewritten to be tightly integrated with the module graph. This enables some nice optimizations, and in particular, allows us to use plain integer identifiers rather than having to hash procedure ids. The downside is that the procedure cache cannot be shared across assembler instances - whether that's a problem in practice I think remains to be seen. I do think that we'll want to further refactor the assembler in the future to enable a greater degree of sharing of the module graph and procedure cache, but there is a tradeoff in performance either way, so I erred on the side of single-threaded use cases for now.
Diagnostics
This PR introduces two dependencies to aid in creating error types and diagnostics,
thiserror
andmiette
respectively. These are using forks I made from the latest versions of both crates, so as to add support for#![no_std]
environments until theerror_in_core
feature stabilizes. But let's take a brief look at how these are used:Defining an error
Let's use the parser as an example here, the following is a subset of it's definition that demonstrates how
thiserror
can be used to define the type, while simultaneously implementingFrom
for any errors it encapsulates, as well as theDisplay
, andError
traits:That's it! It makes defining error types for each task convenient, and provides a very natural way to display the data associated with an error as part of it's
Display
implementation.Now let's extend our
ParsingError
type for use in our diagnostics system:As you can see, this feels like a natural extension of
thiserror
, with support for decorating things that are convertible toSourceSpan
as "labels" in the diagnostic output. Here's what an instance of theUnrecognizedToken
error above looks like when rendered with source code:I actually lifted the example above from our test suite (the
module_alias
test). These pretty errors are enabled by associated source spans with specific errors, and then pairing them with a reference to the source code to which those spans are derived, and this can be done in a few different ways.MASM Serialization
While this PR does not remove serialization of MASM, it does change it, as the AST has changed, and some of the restrictions have been loosened. I think we should plan on removing serialization entirely once the MAST refactoring is done, which should now be trivial with the changes contained in this PR having already set the stage for it.
TODO
There are three things that still need to be done, and they are small, but I want to call them out here:
next
, I'm ~12 commits behind, and there are certainly conflicts there, but I wanted to tackle that once I got this open first.cargo test -p miden-stdlib -- crypto::sha256::sha256_hash_memory
. It is failing withFailedAssertion { clk: <some number of cycles>, err_code: 0, err_msg: None }
, but no additional info. I'm assuming it is due to my changes, but find it odd that all other tests pass and this one does not.begin end
orif.true end
in the surface Miden Assembly syntax. I have chosen not to allow this in the parser, to keep the grammar simpler, and because it honestly doesn't make sense that anyone would write an empty block by hand, or what the purpose of doing so would be. This obviously has nothing to do with supporting empty blocks in MAST, only in MASM source code.TL;DR
This is an enormous PR, but it is more efficient to group these changes together like this than to have made them incrementally (which would have taken many weeks). That said, please take as much time as you need, I don't want this to be a nightmare to review. For that reason I spent an inordinate amount of time breaking this up into smaller commits that can be reviewed piecemeal, but stack in such a way that you can pretty much just start at the oldest commit and work forward and have it go fairly painlessly.
I'm also happy to walk through the changes or any questions you have on a call, in Discord, etc., if you feel it will be easier to have me on hand to answer questions you have. I'll literally pair up for the whole review if you want - I opened this monstrosity, so its only fair I go out of my way to make it easy on the rest of you. Just let me know!
I ran the benchmarks, and here's what I'm seeing compared to
next
: