ship even more of the compiler in source form

[andrewrk]

You know what's really pleasant to work on? The build system. The zig build system is shipped in source form, and when you run zig build, zig compiles the build runner from source, and then runs it. In order to do that, the compiler needs only 2 capabilities: compiling source into a native executable, and executing child processes.

Here are some components that could be shipped in source form rather than compiled into the zig binary:

• [x] zig fmt
• [x] zig reduce
• [ ] translate-c
• [ ] aro
• [ ] package fetching
• [x] Autodoc
• [ ] building glibc stubs
• [ ] building libc++, libcxxabi, libtsan, libunwind
• [ ] building musl, mingw, and wasi-libc
• [x] libc installation detection
• [x] msvc installation detection
• [x] objcopy
• [x] resinator
• [ ] ~all the machine code backends other than the one used to produce native executables~ separate proposal

All of these components have simple interfaces that could be communicated over IPC or the file system. I admit however that last one is a bit spicy.

Implementing this issue would accomplish the following things:

• Significantly decrease compiler build times
• Make it easier to contribute to zig because no compiler rebuild would be needed in order to test edits to any of these components.
• Smaller installation size. Source code is surprisingly compact and compresses extraordinarily well. For example, the xz-compressed zig compiler executable built with -Denable-llvm=false -Doptimize=ReleaseSmall on x86_64 is 2.9 MiB (9.9 MiB uncompressed), while the compiler source code catted together into an xz stream is 1.6 MiB.
• The lazily built executables can take full advantage of native CPU features. We can have AOT cake and eat JIT too!

[VisenDev]

Possibly related proposal

https://github.com/ziglang/zig/issues/18834.

This proposal would allow other zig projects to ship zig source code rather than compiled binaries as well.

[der-teufel-programming]

Shipping current version of Autodoc in source form would require also shipping ZIR and parts of InternPool

[mikdusan]

I like this. It's always irked me that we have things like src/musl.zig that has sort of poor-mans build logic when the build system already exists.

[nektro]

because that logic uses the cache but needs to be callable by build-exe etc. making them use the actual build system is a good idea imo but will require a medium amount of work.

[nektro]

does this proposal being implemented mean for example that zig reduce would start shipping as zig-reduce and/or the zig binary should check PATH for binaries to ensure the space version still works?

[andrewrk]

does this proposal being implemented mean for example that zig reduce would start shipping as zig-reduce and/or the zig binary should check PATH for binaries to ensure the space version still works?

No. It would be the same CLI.

[jayrod246]

So same CLI, and it would re-build the relevant source files if necessary - like how zig build works. Nice!

[paperdev-code]

all the machine code backends other than the one used to produce native executables

This could be very interesting, I imagine specifying a 3rd party package containing a machine backend I need through my build.zig.zon, which could be very good for embedded projects.

[tecanec]

You know what's really pleasant to work on? The build system.

Well, that explains it...

The lazily built executables can take full advantage of native CPU features. We can have AOT cake and eat JIT too!

This is just speculation, but I feel like JIT would synergize really well with comptime. Especially for things like std.simd.suggestVectorLength, whose goal is specifically to automatically optimize for the given hardware.

all the machine code backends other than the one used to produce native executables

This could be very interesting, I imagine specifying a 3rd party package containing a machine backend I need through my build.zig.zon, which could be very good for embedded projects.

I was thinking something similar. Custom backends sound like a big deal for non-mainstream platforms, including proprietary, niche, or even in-development architectures, including experiments and pet projects.

Overall, I am a bit concerned about compile times, though. I know caching is a thing, but it's not perfect.

[ghost]

I'm half-Indian so I think that last bullet could use a bit more heat.

An observation: whatever gets compiled on-demand will have native opt, but whatever we ship as binary won't -- and if one of those things is the compiler itself, then compiling everything on-demand may incur a noticeable delay.

A solution: ship the entire project as source, and provide only a single, tiny binary: the compiler, itself compiled for the absolute bare-bones no-extensions version of the host, with only the host platform backend, but capable of outputting code using all possible features. Upon install, this binary detects the host's features, compiles a copy of itself using native opt, then deletes itself and hands control to the copy.

Now obviously this is pretty wacky: infeasible for the moment, annoying for the first time installer. When it becomes possible though, it will in fact make up the lost time the very first time a serious project is compiled. Also it's just cool to not have a single bit of externally-compiled code in the installed package.

||don't worry, i'm not sticking around. just not quite excised from my own head yet.||