Open andrewrk opened 1 year ago
I see the 1.0.0 milestone for this, is it meant to be a blocker for 1.0? or simply the goal
The milestone, on an issue labeled "proposal", means that a decision must be made to accept or reject that proposal before tagging the release corresponding to that milestone.
For an issue labeled "accepted", the milestone means that it must be implemented by then. So, if this proposal is accepted, then I will evaluate at that time which milestone to move it to.
In the near term, the machine code generated by Zig will become less competitive. Long-term, it may catch up or even surpass LLVM and GCC.
IMO, this is the biggest question. One of the most compelling reasons to use Zig is runtime performance of software written in Zig. Without LLVM's optimization passes, what will that look like?
Long-term, it may catch up or even surpass LLVM and GCC.
We can implement our own optimization passes that push the state of the art of computing forward. We can attract research projects such as alive2 We can attract direct contributions from Intel, ARM, RISC-V chip manufacturers, etc., who have a vested interest in making our machine code better on their CPUs.
Zig will continue to implement optimization passes of its own over time and get faster.
So here the projects that depend on the ability to compile C++ that I currently developing:
zgui
for my game editor in the future. And also some NDA game platform that have C++ only API that will require some C++-to-C glue code to be compiled, but obviously not implemented.
To me, the ability to seamlessly build any C, C++ and Obj-C is a big selling point of the Zig toolchain even if it is behind a optional flag to enable LLVM and Clang when compiling Zig. A part of the hype momentum around Zig is due to that fact.
If this happens, I think I will remove my donation to the Zig Software Foundation.
TL;DR: Lots of libraries in the game development world (closed or open source) require the ability to compile C++.
I think this proposal would hurt the Zig ecosystem more than it would help it, due to several reasons:
Imho, this proposal strongly violates the
Together we serve end users
idea, as the current direction we're heading is a really good unified native build environment based on a single static executable that can serve projects in arbitrar, sizes, shipping compilers for several major languages, a huge ass support for targets and a build system, making work in systems programming fun, even if one doesnt use Zig as a language.
We are on a good way to replace a huge list of tools with a single executable, making contributions to projects build on Zig fun, easy and platform independent.
When this proposal is accepted, in addition to Zig one will need to have the following tools installed:
We can potentially replace all of those tools wit a single, equally powerful executable, making the live easier for all native devs out there
personal projects that would be affected:
I'm still a beginner I believe, but if it is at all worth it for me to give my point of view, Zig's ability to replace all of the build tools mentioned above is a big reason I was interested in Zig in the first place. I struggled a lot with all the different build systems and Zig is a really refreshing breath of fresh air.
I have two personal projects that use zgui heavily and I had planned on continuing.
With Mach engine we have two dependencies that would be very, very painful to remove (would set us back years):
It will be a long time before Zig's SPIRV backend is capable enough to generate non-GPGPU shaders for graphics APIs (if ever, since it would likely require major language changes) - so I don't see a way for us to escape these aside from replicating what these two projects -- LLVM forks -- do on our side.
Every other C++ dependency I believe we could safely escape from.
A positive long-term effect of this change is that it would push us as a community away from wrapping C++ code and towards more pure-Zig solutions.
Many of the comments in this thread are about people using zgui, wwise, zig-gamedev, assimp, and other C++ libraries wrapped with Zig. It gives you a leg up in the short term, but I worry in the long-term that people's gamedev experiences coming to Zig will be 'initially I saw a nice language... then I encountered the guts of the libraries I was told to use were large, clunky C++ codebases'
Also a beginner, but I think being able to use Zig as a C++ build system (which is what I use for all my private C++ projects) is an invaluable feature to me and I believe many other people. The simplicity of Zig, having a compiler and a build system contained within a single executable (with the added bonus of being easily cross platform), is really cool, and it would be kinda unfortunate to see that feature be removed as an effect of removing clang and friends. However, if this does go through, the ability to fallback to generating .bc
files and invoking clang is nice.
One of my friends pointed out that nothing stops you from invoking clang in a build.zig to compile C++ dependencies, even if Zig stops including clang. I wonder how much of a problem that would really be for these projects?
doing that would be an additional dependency without the ease of zigs cross compilation
Hmm, the more I think about this proposal the less I like it. I'm feeling it would be better to make the LLVM backend non-default (that is, switch -fno-LLVM
for -fLLVM
or something) but not remove it. It seems like this would solve most of the issues: the existence of many bugs that are LLVM's fault, and the slow compile speed, aren't strong reasons to remove the LLVM backend so much as make it non-default. It wouldn't solve the issues of the binary size of the zig compiler or difficulties with building it, of course.
I think I can speak for all gamedevs by saying that removing C++ compilation would be a disaster. Too many amazing existing gamedev libraries and tools are built on C++ that disallowing easy use would strangle adoption in that field, as well as complicate existing projects. dear imgui is the obvious example but it's not the only one.
It's easy to imagine that in the long term, this change will push us towards "rewrite it in zig" with all the benefits that would entail, but the downside is that the existing corpus becomes inaccessible; limiting our options heavily even once zigs ecosystem matures
I think llvm is needed until ecosystem of pure-zig library is very very mature and rich.
Yeah we want faster compiling speed and smaller tarball, but not at the risk of losing one-zig-to-rule-all.
I love the ambition of this proposal, but to reiterate what has already been stated, losing c++ compilation would be losing one of the main selling points of zig. I was drawn to zig in part because it reduces the hellishness of depending on c/c++ projects. Zig having a c++ compiler inside it also has the benefit of there being less c++ to have to deal with, less python to have to deal with, less cmake to have to deal with.
On the other hand, the core of this proposal has too many benefits for it to be rejected entirely. I think reducing the scope would be beneficial. How about:
I totally get the desire to get rid of huge third-party dependencies that bring a lot of baggage. There's also something to be said for avoiding an LLVM monoculture in the programming language space.
Even so, I see this as a net negative for users. What drew me to Zig in the first place was the pragmatic approach of acknowledging that there is a world outside of Zig that needs to be interoperated with for the foreseeable future and even providing a best-in-class cross-compilation experience along with that. I built my project integrating Zig build support with the .NET/MSBuild ecosystem on that selling point.
On the whole, I think this proposal would be an unfortunate (if well-intentioned) bait-and-switch, considering the Zig website for a while has advertised this:
In addition, these blog posts drove a lot of attention to Zig in the past:
Just to be super clear: I don't mean to insinuate bad faith or anything of the sort here. But I think it's fair to say that you have to contend with the fact that this proposal would pull features that are not only usable today, but are also prominently advertised.
All that said... assuming this is even remotely practical, maybe there's a potential middle ground: Would it be possible for Zig to continue to use the Clang frontend to provide C-family support, but rip out the LLVM IR lowering and replace it with lowering to ZIR/AIR? (I guess this is more or less how Aro would be integrated too?)
If this could be done, the codegen dependency on LLVM could be killed, achieving at least some of the goals of this proposal. There's probably also no reason to keep LLD support around as long as zld can catch up, so that eventually goes too. And users remain happy. Some of the build and distro woes would remain, of course, but, that's compromise.
I’m fully in favor of making it possible to use Zig without any LLVM components, but I agree with many of the comments here that it’s important for Zig to maintain the capabilities that it currently has in terms of cross-compilation, compiling C/C++ code seamlessly, and generating maximally performant binaries.
These factors are big drivers of Zig’s adoption, and I fear that damaging them (even temporarily as in the case of code generation quality) would seriously hurt Zig’s future.
Personally, I work in the robotics space, where C++ is the dominant language for many libraries and frameworks. I think Zig has a lot of potential in this space, but being able to integrate with existing libraries is absolutely essential for adoption.
(Please ignore my original comment. My use case is unchanged and I temporarily conflated two components.)
I'll start by stating my opinion: the C language frontends are super important to me for Zig to maintain.
I agree with others on the point that Zig being a C/C++ compiler is a big point of attraction that brought me to the language. I started with loving that idea, and ultimately fell in the love with the language, and now I use both. I don't have anything more to add to that that the others above haven't.
I'll add my own personal experience. I have many personal Zig projects, but my biggest one that people tend to know about in the community is my terminal emulator. There are two important dependencies that would be impacted by this:
Harfbuzz - The far and away single most complete cross platform text shaping engine. Text shaping for those that don't know is the process of laying out text, processing things like multi-codepoint emoji into single glyphs, Asian language handling, etc. This is not something you want to really maintain in your own language (i.e. write natively in Zig) because Harfbuzz is so good and so well supported. This is shipped as a single large C++ file with no other dependencies. Without access to Harfbuzz, gaming and manual-GUI applications (not using a mega framework) would surely suffer.
Objective-C for Mac work. Admittedly, this is probably deprecated over the long term since Apple is pushing very hard into Swift. Still, I use Zig's ability to compile Objective-C files to augment my native Zig Mac applications. I've personally written Zig objective-C bindings using the C API, but I also still like to just write some ObjC sometimes which would otherwise take a big mess of Zig/ObjC-runtime code. Again, I think I could find a way around this one, but I think its worth noting for now that ObjC is still a part of [low-level] macOS GUI development.
Andrew, your dislike of C++ is well known! I don't love it either (to put it kindly). If, as an audacious goal, you wanted Zig to lower C++ usage, I think Zig being able to compile existing projects and enable iterative migration away from C++ to Zig is the way to do it. I think if Zig doesn't support C++, the C++ "people" would just avoid Zig altogether.
Main gripe seems to be the need/desire to be able to compile C++ due to the mountain of pre-existing libraries that are already being used in the Zig ecosystem. For this proposal to satisfy people and not to break a huge selling point for Zig, a C++ (and ObjC?) compiler in Zig would have to be up-streamed.
Dunno what madman will be the one to write that tho.
At the very least, any kind of serious effort towards removing LLVM should be done after reaching 1.0.
That said, I don't think divorcing from LLVM itself is a bad idea, if the Zig toolchain itself can slowly grow to replace its functionality, just so long as it sticks around until opting out of it doesn't result in Zig losing out on its existing features.
While I can see this being a good thing in theory I like others have concerns over what it will do more short term. Zig to me is expected to be a highly performant language competitive with languages like C, if it is not performant then it impacts my ability to use it for writing the projects I am making in it because it simply will be a worse option in practice for such things.
Perhaps this could be mitigated if the C backend becomes fully functional to allow projects to compile to C code and have those compile with typical C compilers for when performance is needed, but otherwise I do not think this would be well-advised until some sort of reasonable performance guarantee can be made. Even with the Zig->C->Machine Code process I feel like you'd be losing some optimization potential as no longer would Zig be able to annotate LLVM IR directly and would instead be confined by whatever C can express language wise so that might not even be a foolproof option either (though at the very least it'd hopefully put it on the level of C for most things).
Edit: Apparently I've been told this is what the bitcode support could be used for, I've never used LLVM bitcode stuff myself before but yeah if that's a supported target and still gets focus knowing people will be using it to generate higher quality optimized code until Zig can compete maybe this is less of an issue, albeit a bit more convoluted in how a project would have to be compiled.
Frankly with how insanely complex x86 is and how much work has gone into LLVM over the years I am doubtful Zig would ever reach the same standard of performance. Other larger languages like Rust which aren't even as entangled with C++ haven't tackled this sort of challenge yet either for instance despite the some similar motivation to and more developer resources at their disposal, and to me that is not a very promising sign for its feasibility (though of course Zig could always be the first thing to prove this long-held mindset of LLVM being impossible to replace wrong...).
Also as an aside while I am not as invested in the C++ compilation support as others may be I do think that it'd hurt a lot of projects. Being a gamedev myself losing the ability to use ImGui would be unfortunate as others have mentioned, and personally I also use Tracy in some of my Zig projects which is also C++-based. It wouldn't be too hard to just compile these libraries and link to their binaries (or use a system library I suppose) but still that just makes Zig a bit more pain to interface with this stuff.
Another thing that I don't think has been mentioned/considered yet, if this proposal were to go through, and that were to happen around or after the time that 1.0 is released, it could cause a split in the zig community. Some people who rely on zig's current toolchain might choose to simply stick to an old version of zig so they aren't forced to migrate their codebase. This would harm everyone involved. Users and devs on the old version would miss out on any future features and optimizations, while the other ones, who chose to update, would be unable to use any of these libraries within their own projects without jumping through hoops to do so.
I agree that apart from the mentioned problems with LLVM, it also struggles with the a baggage of legacy code, e.g. in TableGen modules, in how Clang is tied to LLVM compared to the newer compilers that use middle-level intermediate languages, how story of migration from FastISel and SelectionDAG to GlobalISel stalled for many years and isn't really progressing for all supported architectures, and so on and so forth. Using C++ language for writing such a complex piece of software doesn't help either. But even Rust didn't dare to get rid of it just yet. Thus, I think it would be prematurely to do that for Zig either. Long-term it might be a worthy goal, but definitely not in the upcoming 5 years or so, in my opinion. Just my 2c.
I second @yujiri8 here, having optional LLVM target for many years while working on the Zig backends would be a perfect strategy, reducing the maintenance of LLVM parts and providing room for experimentation and optimization of the mainstream targets directly in the Zig code: ARM64 and x86_64, probably RISC-V in the future, if it really takes off. GHC (Haskell) uses the similar approach for at least a decade already, it seems to work for them.
P.S. Why PE format is not in the list for linker?
Maybe the best long-term solution would be to offer some kind of plugin system for the zig compiler.
Pros:
Cons:
At least create a new external project to mantain the goodness of "zig cc c++ bpf..." and builtin platform SDKs .h and cross compilation!!
There are a lot of people worried that loosing C++-support would be a problem.
I am curious how much of those problems would still remain if Aro (or something else) somehow over the year gained C++-parsing support. Would that help 50% of the worried people? 100%? 0%? There is a difference between parsing C++ and making libraries that are binary compatible with it so it might be harder that just parsing it so that is why I am curious!
An alternative idea here might be to try to remove LLVM/clang over time and also add C++-parsing support to Aro (or something else) such that when the switch actually happens, nothing goes away (or at least C++ doesn't go away).
Zig has ended up in a weird situation where on one hand it is a language but then maybe people like it because of how good it is at cross-compiling and mixing languages.
With my limited insight into the project, the fundamental reason for removing LLVM integration doesn't quite make sense to me. All the reasons for LLVM replacements listed here sound like good, valuable improvements that can be implemented without removing LLVM integration. I see that LLVM is "annoying because it's large, slow, and has bugs" - but why remove the option to use it?
An alternative to full removal would be to make LLVM integration less of a priority:
Moreover, the current Zig compiler already has a fully modular structure. LLVM is (as far as I understand it) already completely optional. (The fully-self-hosted path isn't finished yet - but nothing seems to prevent it from working besides time left in the oven.)
Too many hours of work have been put into LLVM. The optimization passes, the IR and everything are really the best on the market right now and are constantly being improved. While I'm sure that zig's own passes could get to an incredibly good state, in the long term, improving LLVM instead is probably still what would provide optimal executable performance, and would also benefit the whole ecosystem.
If dropping LLVM dramatically improves compilation speed, then it could be used during development, for projects that don't require C++ or objective-C, in order to keep a fast development feedback cycle.
Long-term, Zig plans to have Contributor-Friendly IR, or something equivalent, to optimize for low maintenance for supporting many targets that LLVM would never even dream of, for example MOS https://github.com/ziglang/zig/issues/6502.
I'm not sure that's a great argument given there's a fairly stable and performance-competitive port of LLVM to the MOS 6502, actually... It even supports Rust! While it is not upstream, it hopes to be upstreamed at some point in the distant future.
As a bystander, my opinion is: Removing dependency on LLVM, sure; removing any support for LLVM backends, I'm skeptical (though it appears that this may not be the case -> https://github.com/ziglang/zig/issues/13265 ). Even languages which have maintained their own backends for decades, like Free Pascal, are adding LLVM as a target.
And here I was trying to figure a good way to also get zig lldb
going so that a zig toolchain is a one-stop shop for developers.
I'm new to Zig, but the pragmatic acceptance of reality that lead to having 'zig cc' and 'zig c++' come to be is what convinced me to take Zig seriously and consider it as the foundation for a lot work. I'm now reconsidering and likely to put Zig back on the shelf.
But I look forward to seeing how things turn out! I'm pretty sure things will work themselves out for the better eventually.
@slimsag
Many of the comments in this thread are about people using zgui, wwise, zig-gamedev, assimp, and other C++ libraries wrapped with Zig. It gives you a leg up in the short term, but I worry in the long-term that people's gamedev experiences coming to Zig will be 'initially I saw a nice language... then I encountered the guts of the libraries I was told to use were large, clunky C++ codebases'
Isn't it (mostly) the other way round? "Initially I set up my project with good C and C++ libraries, then I wrote my stuff in Zig, and having a polyglot project never caused trouble"? Particularly for a game, the guts of libraries are a rarely visited place and the long term perspective is another project with its own set of libraries.
Oh man, this makes Zig suddenly a lot less attractive because integrated C++ and ObjC support is one of Zig's killer features to me. Without that it's "just another better C".
I don't care about LLVM specifically, but about losing C++ and ObjC capabilities with zig cc
. Pretty much all of my projects integrate C++ libraries (like Dear ImGui) and on macOS some ObjC code for talking to system frameworks. If I need to suddenly deal with another compiler toolchain to compile this code then it becomes the same mess as other compilers that can't handle C++ and ObjC out of the box.
At least consider including a separate clang executable in the standard Zig distribution, and call that from zig cc
, so that we don't loose any functionality from a user's point of view.
Adding to the choir. We are investigating writing CUDA kernels in Zig (yes, it works) and removing LLVM support would be a massive showstopper. LLVM support (and zig cc/c++) is a big reason why we want to invest heavily on Zig.
It also makes Zig the actual escape hatch from C/C++ for HPC. An area in which Rust still struggles to get into in term of penetration.
And here I was trying to figure a good way to also get
zig lldb
going so that a zig toolchain is a one-stop shop for developers.I'm new to Zig, but the pragmatic acceptance of reality that lead to having 'zig cc' and 'zig c++' come to be is what convinced me to take Zig seriously and consider it as the foundation for a lot work. I'm now reconsidering and likely to put Zig back on the shelf.
But I look forward to seeing how things turn out! I'm pretty sure things will work themselves out for the better eventually.
That one is close to my heart and I would rather write a replacement than expose lldb.
Hi Andrew, thanks for your thoughts.
Is there an estimate of how much time you or the core team is spending on LLVM related issues? Can the community help by taking more care of the user issues with LLVM? Writing more guides?
Post 1.0 there is also the question of how to handle LLVM versions. Will Zig need to be frequently updated to work with last LLVM or will it ship with an old LLVM version? Dropping LLVM avoids that question.
Removing LLVM will be a set back in the number of supported target (I'm thinking Nvidia GPU, but I'm sure there are more).
Zig code seems to be easier to optimize than C++ so maybe some custom optimizations could go a long way. Is there someone working on adding some basic optimization passes in Zig?
In general I'm concerned this will set back Zig 1.0 a few years back. Do you think dropping LLVM will help you move faster? Can some of the LLVM associated churn be offloaded ?
@lorenzogatti
Particularly for a game, the guts of libraries are a rarely visited place and the long term perspective is another project with its own set of libraries.
Exactly, it makes little sense to replicate established popular C++ libraries like Dear ImGui or Jolt Physics in another language, it will always make more sense to use them via a bindings shim to benefit from fixes, optimizations and feature development.
Just to add another datapoint: We have found zig cc
to be an excellent cross-compiler for C++, and we currently ship multiple games (like Baba Is You) using it.
To be honest, my impression was that zig cc
was meant to be one of Zig's killer features, so it would definitely be a shame to see it go.
@slimsag
Many of the comments in this thread are about people using zgui, wwise, zig-gamedev, assimp, and other C++ libraries wrapped with Zig. It gives you a leg up in the short term, but I worry in the long-term that people's gamedev experiences coming to Zig will be 'initially I saw a nice language... then I encountered the guts of the libraries I was told to use were large, clunky C++ codebases'
While this is true, there are other incentives than "removing C++ support" for writing better libraries natively in Zig, so I wouldn't worry much about this. It's expected that this early in the life of a language, the main source of libraries are bindings. For starters, Assimp and friends are massive dependencies with a lot of historical cruft and bloat, probably subpar performance, some traits that are known to itch the zig community :)
@kubkon
That one is close to my heart and I would rather write a replacement than expose lldb.
I'm all in favor of that myself and would love to work towards that! lldb was more of a practical matter (liblldb as a place to start). 'zig dbg' could have been the command instead. :D
The complication of python being involved with lldb was kind of mucky anyway. I just thought that it was worth starting from something that exists, learn some lessons, rinse and regroup.
Assimp and friends are massive dependencies with a lot of historical cruft and bloat...
...but also decades of bugfixes and solid support for even obscure asset file formats. An alternative rewritten in another language will most likely take the same time, but more likely will be abandoned before it reaches the same level of maturity.
I'm not a fan at all of each language being its own library ecosystem island. We should strive for more language interoperability and mixed-language projects, not less.
I'm not a fan at all of each language being its own library ecosystem island. We should strive for more language interoperability and mixed-language projects, not less.
Amen brother.
I'm not a fan at all of each language being its own library ecosystem island. We should strive for more language interoperability and mixed-language projects, not less.
Absolutely, my point is that people who want to have pure Zig libraries will always be able to, and that it shouldn't be an argument to drop C++ support.
I guess I'll be staying on the last version before the drop of LLVM, since that's the main feature I wanted from Zig: Cross-compilation of C++ projects using a build.zig. Hoping for a replacement or fork once it starts misbehaving or getting incompatible after some year, to get LLVM upgrades. Having this dual zig/c++ capable compiler with easy interop without writing wrappers, was the best kind of way to get people into Zig. Though, I'm just repeating the sales pitch from the official Zig page now.
This is the beginning of the end of the LLVM 🔥🔥🔥
As others have mentioned, C++ compilation is a major selling point of zig and it would be a shame to lose that. However, there are meaningful benefits to removing llvm from the default distribution.
Perhaps it would be possible to split llvm dependencies into a zig package that can be added to a project's build system using the package manager. This way, C++ compilation and llvm-based code generation could be added to projects that need it, without increasing the size of the core zig binary.
I'm very much against this proposal. Not only will runtime performance suffer greatly without LLVM, but the number of targets will drop significantly too. You'd also lose any sort of C/C++ cross-compilation support, and the ability to compile anything other than C code in a Zig project. Especially those last 2 points are big advantages of Zig, at least, in my opinion.
And now, instead of embracing LLVM and trying to improve its code generation and report bugs, we'll now be competing against it. And that is definitely not the right way of doing things.
In my opinion, the size of the Zig compiler binary doesn't really matter. If it went from something like 1 GB to 100 MB, yes, that would've been significant, but 150 MB really is nothing today. Also, I'd rather have slow compilation speeds but very good runtime performance than the opposite.
And, finally, while I'm not trying to be pessimistic, I don't think you guys will be able to compete with LLVM's code optimization, at the end of the day. LLVM is a massive project that took years to develop and is still being actively improved, while you guys want to start a whole new IR with code optimizations.
So, in my opinion, I don't think this proposal is a good idea. Instead, we should keep things as is and continue to embrace LLVM like we already are.
If there's a way to do it without killing support for the tier 1 targets, I'm all for it.
And here I was trying to figure a good way to also get
zig lldb
going so that a zig toolchain is a one-stop shop for developers.I'm new to Zig, but the pragmatic acceptance of reality that lead to having 'zig cc' and 'zig c++' come to be is what convinced me to take Zig seriously and consider it as the foundation for a lot work. I'm now reconsidering and likely to put Zig back on the shelf.
But I look forward to seeing how things turn out! I'm pretty sure things will work themselves out for the better eventually.
I can definitely see where you are coming from, but dismissing Zig over a proposal seems pretty extreme ...
This issue is to fully eliminate LLVM, Clang, and LLD libraries from the Zig project. The remaining ties to these projects are as follows:
Note that there would still be an LLVM backend for outputting
.bc
files (#13265), but the Zig compiler would lack the capability to compile .bc files into object files. LLVM or Clang would need to be installed and invoked separately for that use case.In exchange, Zig gains these benefits:
Please read my other comments in this issue before replying: