search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

Nix build using flake #162

Closed league closed 2 years ago

league commented 2 years ago

This is a start on specifying a “flake” for building bifrost with Nix. A flake is a format that pins all dependencies to support reproducible builds. This initial version includes github workflow for building with different releases of the nixpkgs tree (containing compilers, python dependencies, etc.) and different versions of python, then running the tests on all those configurations. (This builds on autoconf.) More background: nixos.org, Flakes on Nix wiki, Flakes tutorial from tweag.io

It doesn't yet support GPU builds, though I've gotten it mostly to work. It's a little tricky to integrate because the libcuda.so.1 must come from the host platform, so it agrees with the kernel version and GPU hardware. If the host is itself NixOS it's manageable, but when Nix is being used on top of another platform (e.g. Ubuntu) we can only build against stubs, and then sub in the real libcuda later. For similar reasons, auto-detecting the right GPU architecture during the build seems to be problematic. As long as GPU architecture is an input to the build, it gets hashed into the package signature, but we can't ask what GPU architecture is “from the inside.” Same story as for builtins.currentSystem for the overall architecture/OS tag. Some hints about libcuda on Nix

All this should be solvable (and hopefully useful). I'd like to continue to work on it and tweak it here... so marking this as a draft.

codecov-commenter commented 2 years ago

Codecov Report

Merging #162 (ced5117) into master (657e705) will not change coverage. The diff coverage is n/a.

:exclamation: Current head ced5117 differs from pull request most recent head 38bfbec. Consider uploading reports for the commit 38bfbec to get more accurate results

@@           Coverage Diff           @@
##           master     #162   +/-   ##
=======================================
  Coverage   58.54%   58.54%           
=======================================
  Files          67       67           
  Lines        5727     5727           
=======================================
  Hits         3353     3353           
  Misses       2374     2374

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 657e705...38bfbec. Read the comment docs.

jaycedowell commented 2 years ago

With regards to auto-detecting the GPU arch., would something like building all available archs. be a workable solution? It would take forever to compile but it should be generic.

league commented 2 years ago

I'd like to find a way for telemetry to quietly disable itself if it turns out that $HOME doesn't exist or is not writable. It prevents me from ever doing import bifrost in a nix build environment.

bifrost-doc>   File "/nix/store/mdvjrgc0m3lhcz75hvcpz9lgnis69wbv-python3-3.9.6-env/lib/python3.9/site-packages/bifrost/telemetry.py", line 52, in <module>
bifrost-doc>     os.mkdir(os.path.join(os.path.expanduser('~'), '.bifrost'))
bifrost-doc> FileNotFoundError: [Errno 2] No such file or directory: '/homeless-shelter/.bifrost'

I was toying with generating the documentation from nix (and maybe restoring the auto-update of gh-pages as a workflow action). To generate the python API docs, it needs a python from which it can import bifrost... which can be arranged but the telemetry bombs in that environment.

I guess that whole preamble could be wrapped in a try/catch and on FileNotFound set TELEMETRY_ACTIVE = False and _INSTALL_KEY to a fresh uuid. It would still fail if somebody called enable or disable because you can't unlink or write to _ACTIVE_KEY, but that's probably okay… at least you should be able to import telemetry.

jaycedowell commented 2 years ago

Now that #157 has been merged we should probably close this PR and open a new one that targets master.

league commented 2 years ago

Yep. I may just rebase/squash onto new master... the timeline is pretty confusing with occasional merge commits from autoconf... but ultimately it should just add 3 files.

jaycedowell commented 2 years ago

I think I will try out the nix thing once the new server gets setup.

league commented 2 years ago

I may merge the nix stuff soon, but I feel like it deserves a small section of the README or manual... maybe before next release. I'll add a CHANGELOG entry as a placeholder. Next time you're on qblocks, you might try a quick nix setup like this:

# Install nix
  wget https://nixos.org/nix/install
  sh install
# Log out and back in, or source the script given at the end of the install to set up current shell
# Optional: install our binary cache, maybe another useful tool or so
  nix-env -i cachix ripgrep
  cachix use bifrost
# Some basic nix config: (unfree for nvidia)
  mkdir -p ~/.config/{nix,nixpkgs}
  echo "experimental-features = nix-command flakes" > ~/.config/nix/nix.conf
  echo "{ allowUnfree = true; }" > ~/.config/nixpkgs/config.nix

Even before a git clone, you should be able to do stuff like the following. (If it needs to configure and build it will, but if you hit exactly the configuration that's in cachix, it will just download. Cachix is usually populated by the CI.)