nix-community / poetry2nix

Convert poetry projects to nix automagically [maintainer=@adisbladis,@cpcloud]
MIT License
860 stars 442 forks source link

Suggestion: Deep learning stack test #999

Open Laurent2916 opened 1 year ago

Laurent2916 commented 1 year ago

Similarly to #608, having a unit test for common configurations, such as those for PyTorch, TensorFlow, and other deep learning libraries, would be incredibly helpful. This could include a basic setup that covers dependencies like CUDA, cuDNN, and other NVIDIA drivers.

flake.nix:

{
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs";
  inputs.poetry2nix_pkgs.url = "github:nix-community/poetry2nix";

  outputs = { self, nixpkgs, flake-utils, poetry2nix_pkgs, }:
    (flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
        };

        poetry2nix = import poetry2nix_pkgs {
          inherit pkgs;
          poetry = pkgs.poetry;
        };

        customOverrides = self: super: { };

        my_env = poetry2nix.mkPoetryEnv {
          projectDir = ./.;
          preferWheels = true;
          overrides = [ poetry2nix.defaultPoetryOverrides customOverrides ];
          python = pkgs.python310;
        };
      in {
        devShell = pkgs.mkShell { buildInputs = with pkgs; [ poetry my_env ]; };
      }));
}

pyproject.toml:

[tool.poetry]
name = "scientific-ml"
version = "0.1.0"
description = ""
authors = ["me"]

[tool.poetry.dependencies]
python = "^3.8"
accelerate = "*"
albumentations = "*"
datasets = "*"
diffusers = "*"
dvc = "*"
fastai = "*"
gradio = "*"
gym = "*"
imgaug = "*"
lightning = "*"
lightning-bolts = "*"
onnx = "*"
optuna = "*"
pycocotools = "*"
ray = "*"
tensorboard = "*"
tensorflow = "*"
timm = "*"
torch = "*"
torchaudio = "*"
torchmetrics = "*"
torchvision = "*"
transformers = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
sterlind commented 1 year ago

I started on this in https://github.com/nix-community/poetry2nix/pull/1109 as part of my PR! I haven't taken on that whole enchilada, but I'll definitely be adding things to ml-stack as I work my way through what I need.