self-hosted compiler: ship it!

[andrewrk]

This is a tracking issue for when we can completely replace stage1 with self-hosted in official builds of Zig.

• [x] add debug info to the LLVM backend
  • [x] initial implementation #11085
  • [x] #11095
  • [x] #11096
  • [x] #11097
  • [x] #11145
  • [x] #11158
  • [x] #11160
• [x] #11674
• [x] get compilation errors into an acceptable state
  • [x] #11288
  • [x] #11632
• [x] adjust the default backend to be LLVM until other backends reach full behavior test coverage
• [ ] ~#6025~ async/await/suspend/resume will not be available in 0.10.0
• [x] #6034
• [x] #11159
• [x] #11169
• [x] #11251
• [x] #11259
• [x] all std lib tests passing with the LLVM backend
  • [x] #11898
  • [x] #11897
  • [x] #11857
  • [x] #11856
  • [x] #11853
  • [x] #12029
• [x] #11450
• [x] #11498
• [x] #11767
• [x] #11274
• [x] full test suite passing with the LLVM backend
• [ ] ~check all the TODO comments in stage2 source code and convert relevant ones into issues blocking this one~ #363
• [x] Eliminate use of conditional compilation on zig_backend in compiler-rt, freestanding libc, test runner, and default panic
• [x] use the real start code
• [x] eliminate conditional compilation on zig_backend in the std lib
• [x] Eliminate the other TODO items from generated builtin.zig file
• [ ] work with individual third party projects to make sure the stage2 transition completes successfully
  • [ ] Bun (cc @Jarred-Sumner)
  • [ ] zig-gamedev (cc @michal-z)
  • [ ] mach (cc @slimsag)
  • [ ] River (cc @ifreund)
  • [ ] aquila (cc @nektro)
  • [ ] https://github.com/ominitay/zigvale-barebones (cc @ominitay)
  • [ ] ncdu
  • [ ] https://github.com/ZigEmbeddedGroup/microzig (zig build -Drelease-small)
  • [ ] https://github.com/mattnite/gyro (zig build, good amount of zig and C code in here)
  • [ ] ZLS
  • [ ] https://github.com/sin-ack/zigself
  • [ ] https://github.com/floooh/sokol-zig

[andrewrk]

~When we release zig 1.0.0, we'll actually have the self hosted compiler built and ready to go, passing all tests. Both projects will be maintained until 2.0.0.~

[andrewrk]

There are some compelling reasons to self-host:

• Makes f128 work without needing gnu extensions
• We can write the doc generator in userland
• Makes builtin overflow stuff work without C compiler extensions
• Ability to easily cross-compile the compiler

We still want the bootstrapping process to be simple though. So here's another proposal. We get a self-hosted compiler going right now. It's the official zig compiler. However the C++ implementation must be able to build the official zig compiler. As long as that remains true, bootstrapping is 1 step process.

[andrewrk]

Things I Want to Improve in the Self-Hosted Compiler

Performance and Caching

• Max out performance of machine with thread pool and async I/O
• Pipeline all the work. Split the job up into individual functions that each produce .o files. We'll have LLVM spitting out .o files before the last source file has been tokenized.
• No mutexes. When a coroutine needs a resource that another thread is working on, it yields to another job, getting resumed (in userspace!) when the resource is available (this looks like async/await)
• Multi-layer caching. Cache files, cache AST, cache individual functions
• Establish a file system watch on source files, detecting changes, running through the pipeline (taking caching into account), and atomically update output files in place. The compiler is a long lived process and some of the caching happens in memory.
• Handle temporary out of memory situations with emitting an event that says "waiting for more memory to be available" and it prints how much was needed along with how much the system has available

Representation of Types and Values

ConstExprValue right now has a lot of footguns built into it, and it wastes memory. The new data layout should accomplish these things:

• Use a minimal amount of memory
• Have at the very least runtime safety for wrong union field access and hopefully more compile errors when adding and removing fields.
• In Stage 1, the Type tells how to interpret the Value. In self-hosted, we should divorce these concepts. This should make comptime casting more correctly represented.
• Introduce lazy values. For example, if you do @sizeOf(error), this can create Value that is backed by a LazyComptimeExpr. We could still find out @typeOf(@sizeOf(error)) without causing the lazy expr to evaluate. Once we get to the end of the compilation, we start evaluating all the lazy expressions. If a lazy expression depends on another lazy expression, it gets skipped, and we make a note to start over once done. If all lazy expressions must be skipped, then it's a compile error, and we show the dependency loop.

[phase]

We'll have LLVM spitting out .o files before the last source file has been tokenized.

How will this work with name resolution? You can't compile a file that depends on a file that hasn't been parsed yet.

[andrewrk]

Here's an example:

• we have 2 cores and therefore thread pool size 2
• thread 1 load,tokenize,parse main.zig, which calls foo(), bar(), baz()
• thread 1 scan top level decls and create jobs to analyze foo(), bar(), baz()
• thread 1 analyzes foo()
• thread 2 analyzes bar()
• thread 1 generates llvm code for foo()
• thread 2 generates llvm code for bar()
• thread 1 emits foo.o
• thread 2 emits bar.o
• thread 1 analyzes baz(). baz() calls @import("quux.zig")
• thread 1 load,tokenize,parse quux.zig
• etc
• main thread calls LLD linker on all the .o files

You can see from this example we would get better parallelism if we prioritized analysis of functions since that creates jobs for the pipeline - it would make thread 2 have something to do while thread 1 analyzes baz(). But this should illustrate the idea.

[ghost]

0.3.0 ... seemed so close 😃

[andrewrk]

Yeah. I couldn't make the deadline. 0.3.0 is two weeks away and I think those two weeks can best be spent on:

• Stack traces for windows and MacOS
• Documentation
• Bug fixes

[ghost]

building master always anyway 👍

[ghost]

Something that I would like to see with a self-hosted compiler is the ability to import the compiler as a library within my application to compile and link new code while running. For example, https://github.com/anael-seghezzi/CToy embeds tcc and provides a creative coding environment that does not require a restart when modifying code.

Is such a thing possible already when importing std.build?

[andrewrk]

Something that I would like to see with a self-hosted compiler is the ability to import the compiler as a library within my application to compile and link new code while running.

Unfortunately that's not ever going to be possible, because of the LLVM and clang dependency. Zig compiler ships with LLVM and clang libraries built into the zig compiler. And Zig supports cross compiling for many targets. To import the compiler as a library would require that LLVM and clang were available in source form (written in zig) so that they could be cross compiled for the target. To give up LLVM/clang and code our own code generator in zig would be giving up state-of-the-art optimizations and a very active community of people working on it.

However, the parts of the compiler that do not depend on LLVM and clang are available in the standard library, for example the parser and formatter. std.zig.parse and std.zig.render.

As for a coding environment that does not require a restart when modifying code, see #68.

[marler8997]

Having LLVM/Clang written in C++ shouldn't prevent the compiler from being used as a library. Just like zig programs can link to and use C libraries even though they aren't in Zig.

[Tetralux]

To import the compiler as a library would require that LLVM and clang were available in source form

I think they meant a library as in a DLL, SO or archive, etc.

[momumi]

Something that I would like to see with a self-hosted compiler is the ability to import the compiler as a library within my application to compile and link new code while running.

It would be cool if you could take the idea of a comptime function and extended it to a just-in-time function. So you could take a function fn regex(comptime re: []const u8, str: []u8) bool, give it a runtime-known string for re, and then libzig.so would give you a function pointer to a JIT compiled version that your program could call.

It'd be a pretty niche feature, so I'm not suggesting it should be included. However, I don't think it's been done before in a statically compiled language, so it'd be really unique feature if it wasn't too hard to implement. Although I don't know too much about compilers, so maybe I'm grossly underestimating the difficult.

[yozachar]

Hi, coming from https://kristoff.it/blog/zig-new-relationship-llvm/#when-is-it-going-to-be-ready, I do not understand why https://github.com/ziglang/zig/projects/2 is closed.

[Avokadoen]

@andrewrk I would like to add zig_vulkan to the third party list. It's a gpgpu ray tracing voxel renderer which I am making for my master thesis.

[mlarouche]

I would like to add the following to the third party list:

• https://github.com/zigimg/test-suite (Make sure that the test suite pass)
• https://github.com/wendigojaeger/ZigGBA (just need to build)

Thanks :)

[mattnite]

I would like to add the following:

• https://github.com/ZigEmbeddedGroup/microzig (zig build -Drelease-small)
• https://github.com/mattnite/gyro (zig build, good amount of zig and C code in here)

[leecannon]

I think zls https://github.com/zigtools/zls should be added to the list of third party projects to check.

[sin-ack]

I nominate https://github.com/sin-ack/zigself. While not particularly interesting it does do some funky stuff regarding pointer alignment which might be worth taking a look at.

[kuon]

I would like to add the following:

* https://github.com/ZigEmbeddedGroup/microzig (`zig build -Drelease-small`)

I am now working with zig on some STM32 MCU and binary size is absolutely critical. Is there a binary size comparison/test suite to ensure common uses cases do not regress? I'd be happy to test the firmware code I am working on for that if it can help. Just poke me on zig discord.

[floooh]

Hi, I'd like to add the sokol-zig bindings to the list of thirdparty projects, it doesn't use any fancy Zig features, but it's a mixed C/Zig project where the C code talks to system-level APIs on Windows, Linux and macOS:

https://github.com/floooh/sokol-zig

...when running zig build -fno-stage1 I get a couple of interesting errors which I don't quite understand:

/Users/floh/projects/sokol-zig/src/sokol/app.zig:316:29: error: comptime call of extern function
    return sapp_color_format();
                            ^
/Users/floh/projects/sokol-zig/src/sokol/app_gfx_glue.zig:6:41: note: called from here
        .color_format = sapp.colorFormat(),
                                        ^
/Users/floh/projects/sokol-zig/src/examples/clear.zig:14:33: note: called from here
        .context = sgapp.context()
                                ^
/Users/floh/projects/sokol-zig/src/sokol/app.zig:430:12: error: extern function cannot be generic
pub extern fn sapp_run([*c]const Desc) void;

Cheers!

(PS: sokol-zig bindings might also be interesting to test ABI conformance on various platforms, for instance on macOS for Intel Macs I had to pad small structs (<16 bytes) so that they become bigger than 16 bytes, otherwise there would be corruption when passing or returning structs by values between Zig and C)

[Vexu]

error: comptime call of extern function

Those are caused by stage1 function pointer semantics, s/?fn/?*const fn/ should fix that.

[floooh]

@Vexu thanks! That's getting me a bit further :) I hit a compiler crash later though (and in the stage1 compiler the code compiles using the new function pointer syntax, but I'm getting a weird crash the first time a C callback function is called. I'll put this info into a reminder ticket in the bindings for now, until I have a bit more time digging deeper :)

PS: correction: at least the simple "clear screen sample" works with the function pointer changes in the stage2 compiler! The other, more complex samples currently show various problems, from a compiler crash to "runs but only renders a blank screen".

The stage1 compiler seems to produce broken code for the new function pointer syntax though, when calling into a C callback function I get a hard crash.

Anyway, it looks promising :)

[Vexu]

In stage1 *const fn is a double pointer, if you want your code to work with both versions you need to check @import("builtin").zig_backend.

[andrewrk]

Hi @floooh, thanks for stopping by. I added sokol to the list of projects. Definitely appreciate the testing but it might be best to wait until all those other boxes are ticked before giving it a serious go - I don't want you to waste your time finding issues that we already know about :-)

[andrewrk]

Hello everybody,

The self-hosted compiler is now the default compiler. I have marked the 0.10.0 milestone with the due date of September 21, 2022.

Please take a look at the upgrade guide for help deciding when to upgrade and how to upgrade.

I will personally be working on the existing bug reports that are affecting third party projects, and then continue working with the projects to get them fully upgraded.

[RobertWHurst]

Congratulations @andrewrk! I know you and your team have been working diligently to make this happen for a long time. It's a serious milestone. Nicely done 🎉

[lin72h]

Just witness a historical event, pure awesome

[hako]

major congrats @andrewrk and team for shipping this huge milestone for Zig! ⚡ 🚢

[malcolmstill]

I have a table of function pointers which stage2 tells me to change to const* (which makes sense). But I then use those function pointers with @call(.{ .modifier = .always_tail } and I get the error error: modifier 'always_tail' requires a comptime-known function. Not sure what I need to do to fix that?

See https://github.com/malcolmstill/foxwren/pull/139

[kuon]

Maybe we could create a specific issue template for bug occurring due to the upgrade? I don't think it will be efficient to make this one longer.

[david4r4]

I have a table of function pointers which stage2 tells me to change to const* (which makes sense). But I then use those function pointers with @call(.{ .modifier = .always_tail } and I get the error error: modifier 'always_tail' requires a comptime-known function. Not sure what I need to do to fix that?

See malcolmstill/foxwren#139

Yes, I agree with @kuon. You should discuss it in another issue or in any community channel. Just to give you a hint, basically you can't tail a function pointer call because you have no idea what body the function has.

By the way, congrats to the Zig team and the whole community! :)

[malcolmstill]

@kuon @davidgm94 will do, thanks.