Closed malte-v closed 1 year ago
shlevy
commented
4 years ago My risc-v board crapped out and I haven't been keeping this up to date, sorry. At the time though I was able to run full cross-compiled NixOS in qemu, not too far from native compliation if I was patient enough.
zhaofengli
commented
3 years ago I managed to cross-compile Nix itself for riscv64:
with (import <nixpkgs> {
overlays = [
(self: super: {
coreutils = super.coreutils.overrideAttrs (old: {
# https://github.com/gentoo/gentoo/raw/1070aa3245b38881a050507e365a2c9c60c23534/sys-apps/coreutils/files/coreutils-8.32-ls-restore-8.31-behavior.patch
patches = old.patches ++ [ ./coreutils-8.32-ls-restore-8.31-behavior.patch ];
});
})
];
}).pkgsCross.riscv64;
nix.override {
boost = boost17x;
}Run under QEMU user mode, I would expect most things to behave as if we were compiling natively on RISC-V:
$ rvnix=/nix/store/42mk9y9x26rr06lymdw2f58lkar4x53v-nix-2.3.10-riscv64-unknown-linux-gnu/bin/
$ $rvnix/nix-instantiate --eval -E "builtins.currentSystem"
"riscv64-linux"However, when building NixOS, I received the same error as @malte-v. Digging deeper reveals it's actually from stdenv.cc.
zhaofengli
commented
3 years ago Ok, I think I misunderstood. Of course building natively is not possible without the bootstrap binaries.
When I attempted to build the bootstrap tools myself from x86_64, I got:
cycle detected in the references of '/nix/store/76vjw1asqssgf82vixbay8qfrp6yzvba-gcc-debug-9.3.0-riscv64-unknown-linux-gnu' from '/nix/store/dwsnp73ml283ckjs3fqncrq544ii6k4c-gcc-debug-9.3.0-riscv64-unknown-linux-gnu-lib'Looking at the store paths, indeed I see libraries in the lib output depending on out:
$ strings /nix/store/dwsnp73ml283ckjs3fqncrq544ii6k4c-gcc-debug-9.3.0-riscv64-unknown-linux-gnu-lib/lib/* | grep 76vjw1asqssgf82vixbay8qfrp6yzvba
GNU C++17 9.3.0 -march=rv64gc -mabi=lp64d -g -g -O2 -O2 -O2 -std=gnu++17 -fstack-protector-strong -fno-strict-overflow -ffunction-sections -fdata-sections -frandom-seed=cow-string-inst.lo -fimplicit-templates -fPIC -frandom-seed=76vjw1asqssgf82vixbay8qfrp6yzvba-gcc-debug-9.3.0-riscv64-unknown-linux-gnu --param ssp-buffer-size=4
[...]Those strings are coming from the .debug_str sections in the resulting libraries, so it should be safe to remove them without actually affecting the behavior. The same thing occurs for bootstrap-tools itself, which contains -frandom-seed references to gcc (bootGCC) and glibc:
GNU C17 9.3.0 -march=rv64gc -mabi=lp64d -g -g -O2 -O2 -O2 -fstack-protector-strong -fno-strict-overflow -fPIC -frandom-seed=9h9l9dr78k9blqqq47qp4hix80z3q02r-gcc-debug-9.3.0-riscv64-unknown-linux-gnu --param ssp-buffer-size=4
GNU C11 9.3.0 -march=rv64gc -mabi=lp64d -g -ggdb -O2 -std=gnu11 -fno-strict-overflow -fgnu89-inline -fmerge-all-constants -frounding-math -fno-stack-protector -fmath-errno -fPIC -ftls-model=initial-exec -frandom-seed=zfahziiglvgr2wbqw78l87hnnl9fzbwp-glibc-2.32-10-riscv64-unknown-linux-gnuUnfortunately nuke-refs and remove-references-to don't work in such cases where the references don't contain the path to Nix store itself (otherwise other parts of the files may be damaged). So I did some quick hack and created a dirty copy of remove-references-to. Should we add the functionality to remove-references-to as a flag?
Afterwards, during the tests I encountered two problems:
crt1.o. Looking at the script it seems like -B is added for ${bootstrapTools} but not ${bootstrapTools}/lib. I added the flag and the build continued. Please help me understand why only -B${bootstrapTools} is normally needed where cc-wrapper will add the lib directory as well.cannot guess build type error which I solved by copying updated versions of config.{sub,guess} from pkgs.gnu-config.Then I successfully cross-compiled bootstrap-tools.
My (very ugly) changes so far, which i have pushed to my fork:
riscv64-embeddedEdit: Remove part about prebuilt bootstrap-tools as now I know the reason for the cannot find crt1.o error.
zhaofengli
commented
3 years ago (Deleted my last post as I thought I had some progress, but I don't)
Ok, I'm moving closer to native compilation, but encountered this error when building xz:
/nix/store/dsh0d6paf2m1q5gkfx915wsgvjfnmfb4-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_join'
/nix/store/dsh0d6paf2m1q5gkfx915wsgvjfnmfb4-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_create'This looks very similar to https://github.com/NixOS/nixpkgs/pull/79793#issuecomment-584539014, but the cause is different.
If you expand the full logs you will see that -lpthread was not passed, and the linking succeeds when it's added. But why wasn't it passed in the first place? There are two related flags, -pthread and -lpthread where -pthread implies -lpthread.
From config.log we see that the test for -pthread fails because -pthread also implies -latomic on riscv64 (but -lpthread doesn't), and we do not have libatomic at this stage. GCC is working around this problem by pulling in libatomic for riscv targets (accepted upstream).
The dependency looks like:
builder for '/nix/store/chbhgv849d47irqjw3fvvpp5vyjk4s39-xz-5.2.5.drv' failed with exit code 2
cannot build derivation '/nix/store/y60ywc5p9wihf4azl4p8g9nxfp89gnjj-gettext-0.21.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pnlayr415x453yn64wd88v2rg7pi68ax-binutils-2.31.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/hzih9nh61259h9z41cq57sq86ssywjgb-binutils-wrapper-2.31.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pmbfdbad7993561yq950g1a1zm6v0qka-binutils-wrapper-2.31.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/17q050qysmr7rrg5l94dk8axwd1mqzg7-bootstrap-stage2-gcc-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/x7rilw004qwqxby0l281xq9lmwm0sgwi-bootstrap-stage3-gcc-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/r646vagbwhd6lxsr415hs416j7dsph93-bootstrap-stage4-gcc-wrapper-9.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4m7vbw67xb8qhipi52k3qyf04fmq1c63-gcc-9.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wf79ml4d4shvkkp3w1gh1hypzgdkf6gc-bash-4.4-p23.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/jwmz2ncxp6sc8zbwyyk7vfk1sh7a1pl8-binutils-2.31.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5lb38amp16j26flbwqlb5nka8imr928y-bootstrap-stage2-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/45aycgd90a26738biw2ls972k6d6wm21-bootstrap-stage3-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vskl5vwhrfp9damgc6af2qlycy9pnpb4-bootstrap-stage4-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/yd69kqm426n9mmab1wwfxd6nkkxfmpqy-gcc-wrapper-9.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4d7z1j4f6dxzaskha65zf1wi1wgn5mvc-gdb-10.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/jnfapcfrf82gyh8p7zxc9a514m6f0m8g-glibc-2.32-10.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rx07m3a9jwqv6xa5w5cxwxdzvd4ksrbz-gmp-6.2.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/2n67ss14jm8zdhckmwa4bf5pwhhj0wnx-libidn2-2.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pv5hhxf8ya2j1qpdw623rbrffm0xysj0-nix-2.3.10.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/k3xgq6x7n163ly6kxj14kx4n97cl92p0-perl-5.32.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/nalslbgz6khybdgdilihh2s96dff9bpm-perl-5.32.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pk6f03wcdq9jhpid5m1lm64a5b5d8s09-perl-5.32.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/k8wz4hzs7sdnqpc9yyq1dy1ah7dj485r-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/v4x8ixgjl9h81z992283f8b11f2nc9qw-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/c3vy42nkl1gq11xg9r9ddk6yafvl2hpv-texinfo-6.7.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/y5x4jgww005y0lk7qraws2snxrx3hi1g-nixos-system-nixos-21.03pre-git.drv': 1 dependencies couldn't be builtI don't know what the correct way to solve this is, as a toolchain with libatomic is only built in the later stages. But we don't actually need gettext for our binutils which will be replaced later anyways. Is it possible to drop the gettext dependency from binutils or is there another way around this problem?
Would like to hear your opinions.
flokli
commented
3 years ago cc @Ericson2314 @luc65r
r-burns
commented
3 years ago @zhaofengli Fyi the gcc reference cycles should be fixed by https://github.com/NixOS/nixpkgs/pull/106954
rastertail
commented
3 years ago Any progress on getting this to work? I'm currently in the same boat as the original author of this issue with the same cannot coerce null to a string error, and I am confused by the progress that has been made since.
r-burns
commented
3 years ago Are there more RISC-V specific issues that need to be resolved? The cross toolchain is working now: https://hydra.nixos.org/eval/1648761?filter=riscv&compare=1648406&full=#tabs-still-succeed
Beyond that, NixOS cross-compilation in general is pretty unstable, and will probably continue to be as long as there are no channel blockers for it.
Atemu
commented
3 years ago I think the problem is rather that there aren't a lot of maintainers who need/want cross-compilation.
Making the channels block on cross-compilation issues would only cause headaches I think.
Some sort of health indicator other than "look at hydra builds" might be helpful but that'd probably also just be red all the time.
r-burns
commented
3 years ago Possibly relevant https://github.com/NixOS/nixpkgs/pull/105294#discussion_r578775840
rastertail
commented
3 years ago I've made a bit of progress on my end, but I am still running into tons of issues. My goal is to build a minimal U-Boot compatible root filesystem image for RISC-V. Note that full native compilation is not one of my goals. Here's what I have run into so far:
allowUnsupportedSystem but that does not seem like the right route to go.Here is the configuration.nix I am using, and I am building it using the command nix-build ./nixpkgs/nixos/default.nix -A config.system.build.qcow2 -I nixos-config=$PWD/configuration.nix.
r-burns
commented
3 years ago Hmm, I've run into issues with numactl too. I'll open a PR later to enable it on powerpc64 and riscv since it seems to be supported on both of those platforms.
I don't know what to do about kexec. That looks like a tough one. I'll look into pipewire though
r-burns
commented
3 years ago 
JohnAZoidberg
commented
3 years ago Fyi #96336 (comment)
Yeah I don't understand how @zhaofengli was able to cross compile nix with boost.
I still get the same error just with nix-build -A pkgsCross.riscv64.boost and his instructions.
zhaofengli
commented
3 years ago Fyi #96336 (comment)
Yeah I don't understand how @zhaofengli was able to cross compile nix with boost. I still get the same error just with
nix-build -A pkgsCross.riscv64.boostand his instructions.
For Nix I bumped the boost dependency to boost17x (see my comment). I haven't touched this thing for a while, but you can try my old tree. Hopefully we can get somewhere further before I receive the HiFive Unmatched board later next month.
DieracDelta
commented
3 years ago @rytone it's not clear to me from your configuration: how are you controlling the uboot configuration settings when cross compiling for a riscv target? Suppose I wanted to build just uboot targeting riscv with a custom .config. Do you have a sense for how I could do that?
Relatedly, any idea how to build just opensbi for riscv with nix? (I haven't looked deeply into this; I'm about to do a deep dive).
rastertail
commented
3 years ago @DieracDelta To be completely honest, I'm not totally sure what I'm doing makes any sense, so take this with a grain of salt: What I [think I] am doing right now is building a NixOS image that will be detected by U-Boot where I have built an OpenSBI+U-Boot firmware image following these and these instructions, not at all integrated into the NixOS build. Ultimately, I want to be able to run something like this (I think):
qemu-system-riscv64 -M virt -m 256M \
-bios path/to/opensbi/fw_payload.elf \
-drive file=path/to/nixos.img,format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0Also, related to picking this up again, I am now running into the issues in #116207 when trying to do a fully bootstrapped build, and when just cross-compiling, I run into other random errors depending on what NixOS revision I use.
DieracDelta
commented
3 years ago That's awesome. @rytone could you elaborate more on what you mean by fully bootstrapped?
I'm able to just spin up a nix shell with crossPkgs.riscv64.gcc successfully. For some reason crossPkgs.riscv64.clang won't build; the given error is that llvm7 doesn't exist for riscv. I'm not exactly sure how to override the llvm version.
From that nix-shell, I can build opensbi and u-boot targeting riscv. I'd like to make derivations for opensbi and u-boot that allows for extra targets to be added as well as .config to be tweaked in the case of uboot. I'm not really sure where to start. I'm not familiar with the pre-existing infrastructure for uboot on arm; it seems a bit magical right now.
rastertail
commented
3 years ago Yeah, not sure if bootstrapped is the right term, but its what I went with :sweat_smile:. By "fully bootstrapped" I meant what I have in this configuration where the majority of the RISC-V packages are built under RISC-V virtualization rather than being cross-compiled. The alternative is to import a fresh copy of nixpkgs in the call to make-disk-image instead of inheriting pkgs from the configuration; that way all of the packages are cross-compiled.
With regards to building OpenSBI and U-Boot, I built them like you did just using a shell, though I didn't try using clang at all. I agree that in the long run being able to build them as a Nix derivation would be nicer.
rastertail
commented
3 years ago Quick update: Just realized that while my cross-compilation technique is able to build all of the derivations necessary for a RISC-V NixOS installation, it is unable to completely build the image itself as it tries to run some final setup of the RISC-V image under an x86 VM. I looked and it doesn't seem the VM infrastructure is suited for this sort of compilation... I tried hacking it to spin up a RISC-V VM instead but it got stuck booting. Going forward I would say abandon that idea of cross-compiling everything and instead focus on getting the proper "Nix way" to work.
DieracDelta
commented
3 years ago I got confused with my paths. It turns out I am having trouble getting a nix shell with a gcc cross compiler toolchain with x8664 as my host and riscv64 as my target. It seems to be more involved than just defining a cross system like this:
riscvPkgs = import nixpkgs {
localSystem = "${system}";
crossSystem = {
config = "riscv64-unknown-linux-gnu";
abi = "lp64d";
};
}; And then passing in riscvPkgs.pkgs as a nativeBuildInput to nixpkgs.pkgs.mkShell. This gets me a riscv gcc toolchain that only runs on riscv.
DieracDelta
commented
3 years ago https://github.com/oxalica/rust-overlay/blob/0bb9ef6d8b34e5579d7384201f3106a49ce3deca/examples/cross-aarch64/shell.nix fixed my issue. TLDR is to put riscvPkgs.buildPackages.gcc into the nativeBuildInputs when running mkshell.
For reference, here's a more complete picture of what I'm doing:
let
riscvPkgs = import nixpkgs {
localSystem = "${system}";
crossSystem = {
config = "riscv64-unknown-linux-gnu";
abi = "lp64d";
};
};
in pkgs.mkShell {
nativeBuildInputs = [ riscvPkgs.buildPackages.gcc riscvPkgs.buildPackages.clang ];
};
This gets me both the clang and gcc compilers, both prefixed with riscv64-unknown-linux-gnu-.
Relatedly, any ideas where the gnu-toolchain source is being passed in? I've got a fork that's somewhat diverged from the master branch that I'd like to build. I was hoping to override the src attribute of its derivation, but I'm not finding any derivation when grepping Nixpkgs...
zhaofengli
commented
3 years ago A bunch of commits and hours of waiting later, here's NixOS running on a HiFive Unmatched:
This is my Nixpkgs tree, with most changes submitted as PRs already. A summary of what's required for a bare minimum system closure (checked means it's merged to master/staging):
Unmatched-specific bits are in available in this repo, with more information being added shortly. A binary cache is available here, though I'm still in the process of pushing binaries to it.
At this stage, I really, really want to see #115406 merged in the near future :smile: I've made a simple script to start a riscv64 VM in QEMU using a cross-compiled kernel, in the hope that it will be useful for maintainers without access to RISC-V hardware.
Edit: Note that I did not go the cross-compilation path and everything is built natively (with the help of qemu-user which sometimes does not work). I intend to use the machine as a desktop for a while, which may prove to be difficult due to the lack of a browser :laughing:
To keep track of other PRs:
zhaofengli
commented
3 years ago Relatedly, any idea how to build just opensbi for riscv with nix? (I haven't looked deeply into this; I'm about to do a deep dive).
I have packaged OpenSBI here which I plan to submit to Nixpkgs later.
zhaofengli
commented
3 years ago I just submitted #125451 to add support for building an SD image for qemu-system-riscv64 with U-Boot. If you want to try it out, I have made a pre-built image which you can run with:
# If you have https://github.com/NixOS/nixpkgs/pull/125448
nix-build -A pkgsCross.riscv64.ubootQemuRiscv64Smode -o uboot
# Otherwise, download a prebuilt binary at https://github.com/zhaofengli/unmatched-nixos/releases/download/2021060200/y7qjgkyhxxmqc45j3nnib6y457q72kfr-uboot-qemu-riscv64_smode_defconfig-riscv64-unknown-linux-gnu-2021.04-u-boot.bin
qemu-system-riscv64 --nographic -M virt -cpu rv64 -m 1G \
-kernel uboot/u-boot.bin \
-device virtio-blk-device,drive=sdimage \
-drive file=decompressed-image.img,format=raw,id=sdimageRegarding what is present in my binary cache, here is the entire closure that I'm pushing to it. It includes everything in a minimal installer image, the Sway desktop environment, as well as other fun things (e.g., SuperTuxKart). The binaries are built against my riscv-cached branch.
substituters = https://cache.nixos.org https://unmatched.cachix.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= unmatched.cachix.org-1:F8TWIP/hA2808FDABsayBCFjrmrz296+5CQaysosTTc=Firefox is now running on native RISC-V NixOS. For this I cheated a little bit by using the X86 version of Node.js during build since a native build of node hangs on node_mksnapshot (more investigation needed). It's only used during the build process and does not affect the results.
If you want to try out Firefox and SuperTuxKart on RISC-V NixOS yourself, I put together a new graphical image. With VirtIO-GPU + VirGL (use the included launch.sh), SuperTuxKart is pretty playable with some texture glitches. Now there are also more packages in the binary cache.
Madouura
commented
2 years ago I was planning to get one of the StarFive VisionFive RISC-V boards, I'm surprised there's already been an attempt at this. I'll try my hand and testing some software from the repo and maybe making a list of what works right now on NixOS and what doesn't.
Madouura
commented
2 years ago Got it ordered, should be here in about a week. Any suggestions/requests on what to test first?
fgaz
commented
2 years ago Could you try qbe.tests? iirc it recently got riscv support
Madouura
commented
2 years ago Sure thing. It cross-compiles well, so I'll test it as soon as I get a working test environment on it when it arrives.
Madouura
commented
2 years ago Here's some preliminary footwork: #168826
Madouura
commented
2 years ago Could you try
qbe.tests? iirc it recently got riscv support
nix-build "<nixos>" -A qbe.tests
It appears to work on hardware!
Madouura
commented
2 years ago We're now in nixos-hardware at https://github.com/NixOS/nixos-hardware/tree/master/starfive/visionfive/v1.
YoshiRulz
commented
1 year ago Just got my VisionFive 2, where do I start? Is there a Matrix room specifically for RISC-V?
Madouura
commented
1 year ago I got mine today as well.
I'll be trying to add it to nixos-hardware when I get a chance.
0x4A6F
commented
1 year ago Matrix Room: Exotic Nix Targets is used for RISC-V.
NickCao
commented
1 year ago Homemade (without using the starfive proprietary tools) firmware for visionfive2: https://hydra.nichi.co/job/nixos/riscv/firmware-vf2/latest/ (u-boot hasn't been adapted to nixos standards yet)
skeuchel
commented
1 year ago Can anyone interested look at https://github.com/NixOS/nixpkgs/pull/221306 to make gn compile on RISC-V?
fgaz
commented
1 year ago Star64 nixos-hardware pr: https://github.com/NixOS/nixos-hardware/pull/662
And here's an image I built directly on the board: https://app.cachix.org/api/v1/cache/fgaz/serve/hj72i7znp0rvc4dxs2xslw6b3ik3wifb/sd-image/nixos-sd-image-23.11pre-git-riscv64-linux-pine64-star64.img.zst (8GB RAM, user/pw: nixos)
fgaz
commented
1 year ago To build the image on buildPlatform=riscv64, these two patches are needed: #240348 #242019
nyabinary
commented
1 year ago @fgaz Both merged now
fgaz
commented
1 year ago @Nyabinary yes I have working images here: https://sr.ht/~fgaz/nixos-star64
fgaz
commented
1 year ago And we have a new matrix room at #riscv:nixos.org
samueldr
commented
1 year ago Is there any reason to keep this issue open still?
I fear it's going to become an unactionable grab bag of comments and unrelated issues.
AFAIK, NixOS kinda just works, as much as any other similarly-tiered platforms at the moment, no?
RaitoBezarius
commented
1 year ago Correct. There's no further reason to keep an issue much more than "wow, look how NixOS came to be on RISC-V". We can move on to more actionable fine grained issues when it pertains to RISC-V support in NixOS.
Issue description
Judging from the screenshot in https://github.com/NixOS/nixpkgs/pull/36187, @shlevy has already been successful in running some form of NixOS on RISC-V. I'd just like to know what the status of this is, as naively trying to build an ISO installer failed:
There seems to be some issue with Perl. I guess https://github.com/NixOS/nixpkgs/commit/306d5cdf03ad6375861a20b8885ef38699ca3c23 addressed this, but now it's broken again. I'm still pretty much a biginner to Nix(OS) and unfortunately have no idea how to fix it :( Maybe we could get NixOS on RISC-V running again? I don't know how much work it would be though, since https://github.com/NixOS/nixpkgs/pull/36187 got merged 2.5 years ago and a lot has happened since then.