CUDA-Team: increase Nixpkgs adoption within the scientific computing community

[ConnorBaker]

Increase Nixpkgs adoption within the scientific computing community

This issue tracks the feature-level work to increase adoption of CUDA-related Nixpkgs packages within the scientific computing community.

• [ ] Speak at conferences and meetups to increase awareness of Nixpkgs' scientific computing capabilities.
• [ ] Work with the NixOS marketing team to reach out to industry partners and better communicate the value of Nixpkgs to the scientific computing community.

[ChrisPattison]

One of the things that might be helpful to increase uptake is to have an easily discoverable central page showing how to enable platform specific optimizations for common packages. I think I've seen one for BLAS, but emphasis on easily discoverable/centralized.

As it stands now, I don't think an end user in HPC would be able to easily find all the necessary pieces to build a flake for their codebase, add a nixpkgs overlay for customized build flags, how to run the code they built with their flake **, etc. My personal experience with nix was a very slow learning curve trying to figure out what the relationship was between flakes, nixpkgs, nix the package manager, nix the language, and nixos. HPC end users typically have to get something up and running very quickly for it to be worthwhile, so maybe the community would benefit from a well written blog post here

** This I found confusing at first personally

[ConnorBaker]

One of the things that might be helpful to increase uptake is to have an easily discoverable central page showing how to enable platform specific optimizations for common packages.

Agreed!

As it stands now, I don't think an end user in HPC would be able to easily find all the necessary pieces to build a flake for their codebase, add a nixpkgs overlay for customized build flags, how to run the code they built with their flake **, etc.

Yeah, that's definitely been a pain point. Beyond just setting the BLAS provider to something fast, typically want to enable optimizations for Python which are unset by default in Nixpkgs (which results in rebuilding... a lot of Nixpkgs). And then on top of that, set flags to build for a more recent target (like with AVX2/AVX512 support) or a specific GPU.

I'm hoping to get something that can serve as a tutorial and a minimal working example up at some point. Would also be good to get it in CI.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cuda-team-roadmap-and-call-for-sponsors/29495/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cuda-team-roadmap-update-2023-08-29/32379/1

[PhDyellow]

Are you aware of BioNix? https://academic.oup.com/gigascience/article/9/11/giaa121/5987272?login=false

@jbedo is maintaining BioNix and the R modules here in nixpkgs.

[jbedo]

FWIW there's also an Nix HPC matrix room. There's definitely some HPC specific issues, however I've found they're usually pretty specific to the HPC facility. Documenting the more common ones and work arounds would be useful though.