search

ziglang / zig

General-purpose programming language and toolchain for maintaining robust, optimal, and reusable software.
https://ziglang.org
MIT License
33.94k stars 2.48k forks source link

Tier 1 FreeBSD Support for x86_64 #1759

Open andrewrk opened 5 years ago

andrewrk commented 5 years ago

Here's the issue to track FreeBSD support.

History:

Checklist:

Once this is complete, we can mark FreeBSD x86_64 as having Tier 1 support.

bnoordhuis commented 5 years ago

get my raspberry pi (aarch64) hooked up to the CI somehow

That's a raspberry pi 3? We have a small armada of those in the Node.js CI. They're a lot faster than the pi 1 and 2 but still so much slower than regular server hardware that we cross-compile and shard the test suite across 10 or 12 machines.

That's a long-winded way of saying they're not really an option unless you're really patient. :-)

andrewrk commented 5 years ago

That's a raspberry pi 3?

Yes it is. Thanks for the heads up!

valpackett commented 5 years ago

re: CI, I'm "starting" a buildbot cluster at https://porting.club — currently it's just a Scaleway 4-core ThunderX VPS (the master as well as the first worker, which is a jail on the same box). The "plan" is to have a cluster with many different OSes and CPU architectures specifically for stuff that has portability needs — compilers/interpreters/VMs, libraries that touch OS specific stuff, etc. The idea is to not use throwaway containers or VMs, but instead set up persistent environments — so that they can be used not just for CI tests or nightly builds, but also for development (i.e. project developers should be able to ssh into the workers).

(btw ThunderX cores are comparable to Cortex-A72s / significantly faster than the A53s the RPi has)

(@bnoordhuis I guess Node.js builds its own copy of V8, while Zig wouldn't need to build any monster dependencies, and the compiler itself is kinda lightweight. Oh wait, LLVM can be a memory hog though, and the Pi only has 1GB.)

andrewrk commented 5 years ago

@myfreeweb that's exciting. Keep me updated.

In other news I updated the support table in the README to use a "tier" system: https://github.com/ziglang/zig#support-table

I think what I'll do is review the freebsd2 branch, make any changes that I see fit, and then merge it into master. Then freebsd becomes Tier 2 for x86_64 and i386. And then this issue can remain open to track Tier 1 support.

andrewrk commented 5 years ago

I just merged the freebsd2 branch into master, and so now FreeBSD has Tier 2 Support. This issue is tracking the effort to achieve Tier 1 Support.

ararslan commented 5 years ago

FreeBSD is now available on Cirrus CI. I set it up on one of my repositories to test it out. As an example, you can see the config here and a build log here.

andrewrk commented 5 years ago

I rented a freebsd server for an hour and poked at it. Here's what I accomplished:

Now Hello World works and the behavior tests pass in debug mode. ReleaseFast mode doesn't pass, and the standard library tests require more porting work in order to compile. But they're in a state where I think it's easy to contribute to.

andrewrk commented 5 years ago

One of the developers from FreeBSD stopped by and clarified that the stable kernel ABI is in fact through libc, and so we will now switch to doing it that way, like we do for MacOS.

andrewrk commented 5 years ago

Thanks @ararslan and @mgxm for the CI pull requests. We now have sr.ht building successfully on x86_64 FreeBSD on every pull request and master branch commit. However, all tests are disabled except for the behavioral tests in debug mode (which all pass).

So - we've now ensured that FreeBSD won't regress accidentally. And we can progress on Tier 1 Support by getting more tests to pass, and then making sure those tests are run by CI. Once we get all the tests passing, I'll work on static FreeBSD builds being available on ziglang.org/download.

Progress!

andrewrk commented 5 years ago

See this comment for a proof of concept of cross compiling glibc: https://github.com/ziglang/zig/issues/514#issuecomment-470957284

With regards to FreeBSD here are the next steps:

valpackett commented 5 years ago

observe the link options. Find out what startup files are needed

 "/usr/bin/cc" -cc1 -triple x86_64-unknown-freebsd13.0 -emit-obj -mrelax-all -disable-free -main-file-name basenamewtf.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/7.0.1 -fdebug-compilation-dir /home/greg/src/localhost -ferror-limit 19 -fmessage-length 284 -fobjc-runtime=gnustep -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/basenamewtf-eff904.o -x c basenamewtf.c -faddrsig
 "/usr/bin/ld" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o basenamewtf /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/basenamewtf-eff904.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o

keep in mind the funny build process for crt1 required for the ABI version metadata

crtbegin.o and crtend.o are unnecessary

hm. Do ctors/dtors from shared libraries work without crtbegin/end in the executable?

andrewrk commented 5 years ago

funny build process

I've been staring at glibc code for 7 days now, that "funny build process" looks incredibly clean and easy to understand :rofl:

hm. Do ctors/dtors from shared libraries work without crtbegin/end in the executable?

Based on my understanding, crtbegin.o and crtend.o are the "old way" of doing things and aren't necessary with newer C runtime libraries. I haven't tested it yet though.

Another option we have is saying "that's not supported". I think you would have trouble finding someone who thinks shared library constructors/destructors are a good idea.

Finally a third option is to stop fighting it, and support building crtbegin.o and crtend.o. I looked at the code in gcc, and it wouldn't be straightforward. But I have a few ideas to try that could solve it.

andrewrk commented 5 years ago

By the way what ABI does zig say is native for you on freebsd?

zig targets

Which one do you think makes sense to be the native one? I'm trying to understand this triple: x86_64-unknown-freebsd13.0 and I wonder if FreeBSD has a patch to clang to add another ABI?

daurnimator commented 5 years ago

I think you would have trouble finding someone who thinks shared library constructors/destructors are a good idea.

✋ why would they be a bad idea? How else should a shared library e.g. initialise an initialisation mutex?

andrewrk commented 5 years ago

How else should a shared library e.g. initialise an initialisation mutex?

like this

valpackett commented 5 years ago

crtbegin.o and crtend.o are the "old way" of doing things and aren't necessary with newer C runtime libraries

oh, right, ctors/dtors are the old way, and the new way is init_array/fini_array. (And FreeBSD on new architectures like aarch64 doesn't support ctors/dtors at all.) I always forget which one is the new one lol. Seems like the _arrays are handled by rtld, so we're fine here!

I'm trying to understand this triple: x86_64-unknown-freebsd13.0 and I wonder if FreeBSD has a patch to clang to add another ABI?

Look at the Triple class in LLVM, everything is parsed with StartsWith and the version is parsed into numbers.

Nothing prevents me from doing

cc -cc1 -triple x86_64-unknown-freebsd69.420 …

:)

andrewrk commented 5 years ago

I started working on making FreeBSD builds available in the llvm8 branch. I ran into a snag. This command:

release/bin/zig build-exe ../example/hello_world/hello_libc.zig --library c

when I run it manually in a freebsd server that I'm renting, works fine. But in the sr.ht freebsd CI, it prints:

/usr/home/build/zig/example/hello_world/hello_libc.zig:1:11: error: C import failed

and nothing else. It's hard to tell what the problem is because as far as I understand the environments should be the same, and it works fine in the environment that I have shell access to.

andrewrk commented 5 years ago

FreeBSD binary builds available on the download page are also blocking on this sr.ht issue.

andrewrk commented 5 years ago

The sr.ht issue is resolved and I have pushed a commit to master branch enabling FreeBSD CI builds to automatically go on the download page when tests pass.

andrewrk commented 5 years ago

FreeBSD builds are now available on https://ziglang.org/download/. We still don't have all the tests passing yet though.

andrewrk commented 5 years ago

New TODO item for FreeBSD: audit the structs in std/os/freebsd.zig. See https://github.com/ziglang/zig/pull/2326/files#r279576473. Some of the structs were copied from std/os/linux.zig rather than system header files, which results in ABI mismatches. They're really annoying to troubleshoot, and there are probably other mistakes as well.

dch commented 5 years ago
andrewrk commented 4 years ago

Good news for FreeBSD folks, I added some memory profiling code in #3482, and it looks like there are some straightforward changes to make which will greatly reduce memory usage. Hopefully this will put us within sr.ht's requirements so that we can enable all the tests. Then Tier 1 Support will basically be down to supporting cross compiling for FreeBSD's libc.

andrewrk commented 4 years ago

More progress: #4458 All the test suite is enabled in the CI now except for std lib tests, which are hitting OOM. However, stage1 has seen a handful of improvements recently regarding memory consumption, and we're only a couple hundred mebibytes away from all freebsd tests passing on sr.ht's 4 GiB VM image.

The-King-of-Toasters commented 2 years ago

I'm going to audit the structs today, will post any updates here if they arise.