Runtime code generation

[boggle]

zigs cross-platform focus makes it appealing for writing compilers or as IL for compilers.

What would it take to add runtime code generation to zig, i.e. a facility for explicitly constructing zig ast (either by parsing string literals or programmatically, using some API), compiling such an ast in-process, and running the result? Is anything like that on the roadmap?

Even if execution isn't immediately feasible for all targets, compilation to e.g. webassembly follwed by interpretation might still be useful.

[ghost]

That should be possible by simply using the self-hosted compiler as a library. Or do you have some more specific functionality in mind?

[boggle]

Great that the compiler is a library but a little help can go a long way. How much support syntax/utility functionality would be needed to make this as nice as say evaluating an symbolic expression in lisp? What would be needed so that the compiled code can partially depend on state available in the context?

Ideally Im looking for something similar to what one can do with staging in scala or runtime code evaluation in Julia or lisp, including

• turn a string into a syntax object
• literal syntax objects with quasi quotation
• compile a syntax object to a (possibly parameterized) code object
• dynamically execute a code object
• ideally this is typed (tracks argument and output type)

with the goal of making it easy to write a code generator by simply composing syntax objects

[tauoverpi]

I guess #68 is related and incremental builds since it practically implements half of this with the rest being an interface issue.

[boggle]

Yes, hot code loading is one half of this. The other half is type system and syntax support. Scala 3/dotty supports staging like so: https://dotty.epfl.ch/docs/reference/metaprogramming/staging.html (Paper: https://biboudis.github.io/papers/pcp-gpce18.pdf)

[ghost]

"Code generation" covers quite a lot of diverse territory, and each application has different requirements and ergonomics. In the interest of not getting them all mixed up, here are some personal thoughts on each of them. I hope this is not too far off topic.

1. Precomputation and code specialization for performance
   • related to templates and constexpr in C++
   • related to Jai's #run and #bake
   • partly related to libraries that generate specialized C code for performance (like parser-generators and perfect-hash generators)
   • Adequately addressed with Zig's comptime and inlining directives. Arbitrary compiletime code execution and argument inlining would expand the scope further.
2. Compiletime metaprogramming for custom syntax sugar and embedded languages
   • related to Lisp (and Julia) macros, quasiquotation, etc.
   • can be quite neat - e.g. for an embedded query language with concise notation
   • Almost certainly not going to happen in Zig, since Zig is strongly in favor of explicit semantics and strongly against hidden control flow. Zig does not even allow operator overloading for these reasons.
3. Runtime metaprogramming for ultimate performance
   • related to multistage programming (e.g. in Terra, and now apparently Scala)
   • related to generated functions in Julia
   • related to auto-tuning numeric libraries like FFTW
   • This is something that I would really like to see implemented in Zig, but I haven't the sligthest clue how :stuck_out_tongue:
4. Language backend (compiler as a library)
   • via text generation or direct AST manipulation
   • similar to compiling to C or to LLVM IR
   • very verbose compared to syntactic macros, but unlimited flexibility
   • Zig will probably be usable for this once the self-hosted compiler is complete, but it is yet to be seen whether it will be also good at (both it terms of API usability and quality of generated code).
5. JIT backend
   • similar to previous point, but with slightly different trade-offs regarding compilation time, optimization and linking
6. Interactive use / embeddable scripting language
   • for use in a stand-alone or embedded REPL
   • like Lisp or Lua
   • Supported in principle with hot code reloading. However, Zig is not very suitable as a language since it is strictly typed and not garbage-collected.

In summary, compiletime code specialization is already supported, although some capabilities could still be added. Runtime specialization would be interesting, but I'm not sure what it would look like. Syntactic metaprogramming probably goes against the spirit of Zig. Zig as a backend: probably yes, but too early to tell, since the stage 2 is still pretty new. Zig REPL / embedded Zig: can be done, but unclear benefits.

[andrewrk]

There are no plans to make zig anything beyond an Ahead Of Time compiler.