lerno
commented
1 month ago Ok, so keeping the small output small is super important. What you're seeing here from the stdlib is a combination of things:
-
Some functionality like int128 math functions are supposed (by LLVM) to be available from some external library. Typically its compiler-rt, but I wanted C3 unencumbered by those functions, so in order to get them they're implemented in C3. However they're implemented with as
exportand consequently aren't stripped from the binary at all times. This can be improved further, but should probably be done in conjunction with investigating when they're actually needed by LLVM output. -
There are symbolic stack traces, even if the debug information isn't there, there's still a bit of standard library to handle it so that hitting asserts have some reasonable behaviour. But this also increases the binary footprint.
-
When you hit a direct
unreachableorpanicin the standard library, then the message there is stored as a string. This actually stacks up quite a bit. Partly this could be deduplicated. It's already done per module, but not cross module. Obviously there are even more messages with safety turned on, since the asserts will store a message for where they hit. -
Here's the tricky one: dynamic methods. These are stored in a separate segment on MacOS, or in startup methods directly on Windows and Linux. This is used for example to enable adding custom printf formatting to any type. However, unless we do a single module compilation we don't know if other object files will reference these, so we need to pessimistically keep them. By default methods are stripped for unused types, but anything using the standard library to print will touch on the
Fileat least which has some dynamic methods. Also the allocators do the same. -
Currently the standard library is just provided as part of what's compiled. Stripped for only used code yes, but still provided. And this is adding to the executable, as if we would have statically linked libc every time. So what can be done? Well obviously the standard library could be made into a dynamic library, then that would be invisible, even though the code is there. Another thing would be to allow partially disable parts of the library that always is available at startup, like the temp allocator. More research should go into this than the one I've spent in terms of where the memory goes.
Regarding memory size. I was able to compile examples for WASM4, which has 64 kb of memory. I still retained quite a bit of memory after loading the program. This was some time ago though, and I don't know if this regressed. This was however setting no linking libc, which in itself disables large parts of the standard library. There is also a memory environment settings to use.
All in all, improving this is a work in progress. The more data I get, especially on different platforms, the better I can do.
(BTW that --panic-msg=no changed something indicates that the -Oz setting has a bug)
data-man
lem0nify
Reducing binary size with stdlib
Here is the simplest program in C3:
Let's try make it smaller:
-152 KB. Pretty cool but still huge executable for such tiny program. Let's also strip it.
Still very large. By the way, we use only one procedure from stdlib. Yeah, maybe it has some dependencies but do they indeed take 300KB? Just imagine how much logic a 300KB binary code may contain. It seems the entire stdlib is linked statically into our executable. Perhaps LTO will help us decrease its size more?
No, it does not. Let's see what else the compiler will allow us to do... Safety? Who needs safety?
Well, another decent reduction. Let's also remove panics.
And I have a question here. QUESTION: In
c3c --helpabout optimization level we can read that code is unsafe starting from-O2and panic messages are disabled starting from-O4:Then why do
--safe=noand--panic-msg=noflags still affect output size if-Ozis already specified? 🤔Building without stdlib
So, we finally have 190KB executable. It is 306KB less than on our first step and looks like this is the smallest size we can achieve by linking with the standard library. Let's drop it out too:
Wait, what? No difference? Yeah, it definitely links to entire stdlib regardless which procedures we are using from it. Okay, there is another command line option:
Finally, size comparable to C output executable sizes!
So the C3 standard library is 175KB. You might say that's very little, and you'd be right! But the language will evolve, and its standard library will likely grow. And if it continues to link the entire stdlib into every executable, regardless of whether LTO is enabled, the difference between a binary with stdlib and a binary without stdlib will become larger and larger. But what is a programming language without a standard library? What reason should I have to write C3 without a standard library, if I can continue writing C with a standard library and have binaries of the same size or even smaller? Yes, of course, you might say that I can use any C libraries (including libc) in a C3 program, but I'll have to write a binding first (declare all the functions and structures I want to use from there).
Where am I going with this?
We have plenty of "C killers" for complex high-level programming, such as C++, Rust, Go, etc. They are quite feature-heavy, yes, but if I need to develop a program for which size does not matter, most likely it will be a complex massive system, so I would rather take Rust and fight with a borrow checker than write it in a very low-level language, afraid of making a mistake that will be difficult to debug later.
But we do not have a single "C killer" that is well suited for a driver developer, a hacker writing a rootkit, a developer for microcontrollers, a mobile developer who cares about users of old smartphones with limited memory, a GameBoy Advanced developer, in the end. Or even just an indie developer who is ideologically determined that his simplest program should be tiny.
When I saw C3, the first thing I thought was: "Oh, finally a really low-level nice replacement for C, with nice modern features, but not overloaded with them!" But when I saw the size of the helloworld binary which is, I remind, 496KB on my PC, I was a bit shocked. For reference, helloworld in Rust in the release build without any additional size optimizations is 411 KB on my PC.
Sorry for this huge post and my cry from the heart, but I implore you, C3 dev team, to implement LTO or something, so that we can use parts of the standard library in super-low-level programs without blowing them up to hundreds of kilobytes, and generally think about ways to reduce the size of the binary. Size, damn it, is not as unimportant as many people think!
Thank you for your attention! ❤️