julia.withPackages fails when installing Flux

solna86 commented 1 week ago

Describe the bug

Instantiating julia with the package Flux fails when resolving dependencies.

Steps To Reproduce

Steps to reproduce the behavior:

nix-shell -p 'julia.withPackages ["Flux"]'

Expected behavior

Should instantiate julia with a Flux package precompiled.

Additional context

It seems that there is a problem with dependency resolution, but I'm attempting to install Flux just for CPU:

ERROR: Unsatisfiable requirements detected for package AMDGPU [21141c5a]:
 AMDGPU [21141c5a] log:
 ├─possible versions are: 0.1.0-0.9.6 or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-0.9.6
 ├─restricted by compatibility requirements with Flux [587475ba] to versions: 0.9.3-0.9.6 or uninstalled, leaving only versions: 0.9.3-0.9.6
 │ └─Flux [587475ba] log:
 │   ├─possible versions are: 0.4.1-0.14.16 or uninstalled
 │   └─restricted to versions 0.14.16 by an explicit requirement, leaving only versions: 0.14.16
 └─restricted by compatibility requirements with LLVM [929cbde3] to versions: 0.5.0-0.9.1 or uninstalled — no versions left
   └─LLVM [929cbde3] log:
     ├─possible versions are: 0.9.0-8.0.0 or uninstalled
     ├─restricted by compatibility requirements with CUDA [052768ef] to versions: 1.5.0-7.2.1
     │ └─CUDA [052768ef] log:
     │   ├─possible versions are: 0.1.0-5.4.2 or uninstalled
     │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-5.4.2
     │   ├─restricted by compatibility requirements with cuDNN [02a925ec] to versions: 4.0.0-5.4.2
     │   │ └─cuDNN [02a925ec] log:
     │   │   ├─possible versions are: 1.0.0-1.3.2 or uninstalled
     │   │   └─restricted to versions * by an explicit requirement, leaving only versions: 1.0.0-1.3.2
     │   └─restricted by compatibility requirements with GPUCompiler [61eb1bfa] to versions: 5.0.0-5.4.2 or uninstalled, leaving only versions: 5.0.0-5.4.2
     │     └─GPUCompiler [61eb1bfa] log:
     │       ├─possible versions are: 0.1.0-0.26.6 or uninstalled
     │       ├─restricted by compatibility requirements with CUDA [052768ef] to versions: [0.3.0-0.7.3, 0.8.1-0.10.0, 0.11.1-0.12.9, 0.13.3-0.14.1, 0.15.1-0.21.4, 0.24.0-0.26.6]
     │       │ └─CUDA [052768ef] log: see above
     │       ├─restricted by compatibility requirements with AMDGPU [21141c5a] to versions: [0.4.0-0.5.5, 0.7.0-0.14.1, 0.15.1-0.17.3, 0.19.0-0.19.4, 0.21.0-0.26.6], leaving only versions: [0.4.0-0.5.5, 0.7.0-0.7.3, 0.8.1-0.10.0, 0.11.1-0.12.9, 0.13.3-0.14.1, 0.15.1-0.17.3, 0.19.0-0.19.4, 0.21.0-0.21.4, 0.24.0-0.26.6]
     │       │ └─AMDGPU [21141c5a] log: see above
     │       └─restricted by julia compatibility requirements to versions: 0.22.0-0.26.6 or uninstalled, leaving only versions: 0.24.0-0.26.6
     ├─restricted by compatibility requirements with AMDGPU [21141c5a] to versions: 2.0.0-7.2.1
     │ └─AMDGPU [21141c5a] log: see above
     ├─restricted by julia compatibility requirements to versions: 4.0.0-8.0.0 or uninstalled, leaving only versions: 4.0.0-7.2.1
     ├─restricted by compatibility requirements with Metal [dde4c033] to versions: 6.0.0-6.6.3
     │ └─Metal [dde4c033] log:
     │   ├─possible versions are: 0.0.1-1.1.0 or uninstalled
     │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.0.1-1.1.0
     │   ├─restricted by compatibility requirements with Flux [587475ba] to versions: 0.5.0-1.1.0 or uninstalled, leaving only versions: 0.5.0-1.1.0
     │   │ └─Flux [587475ba] log: see above
     │   └─restricted by compatibility requirements with GPUCompiler [61eb1bfa] to versions: 0.5.1-1.1.0 or uninstalled, leaving only versions: 0.5.1-1.1.0
     │     └─GPUCompiler [61eb1bfa] log: see above
     └─restricted by compatibility requirements with GPUCompiler [61eb1bfa] to versions: [6.2.0-6.6.3, 7.1.0-8.0.0], leaving only versions: 6.2.0-6.6.3
       └─GPUCompiler [61eb1bfa] log: see above
Stacktrace:
 [1] propagate_constraints!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; log_events::Bool)
   @ Pkg.Resolve /nix/store/5ka9hc3bpy0881kh6f25kshh7g87lza8-julia-1.10.4/share/julia/stdlib/v1.10/Pkg/src/Resolve/graphtype.jl:1072

Notify maintainers

@nickcao @joshniemela @thomasjm

Metadata

system: "x86_64-linux"
host os: Linux 6.11.1, NixOS, 24.11 (Vicuna), 24.11pre687768.27e30d177e57
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.18.8
channels(username): ""
channels(root): "nixos"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

The issue seems present both in 24.05 and unstable. For many other Julia packages, julia.withPackages works absolutely fine. It is a great addition to NixPkgs.

thomasjm commented 1 week ago

I haven't had time to dig into this yet, but there was an outstanding issue related to CUDA/hardware specific libraries in general, discussed in the original PR here: https://github.com/NixOS/nixpkgs/pull/225513#issuecomment-1802480247.

However, it looks a bit like a straightforward version incompatibility. One thing to try might be to bump the General registry.

solna86 commented 1 week ago

I haven't had time to dig into this yet, but there was an outstanding issue related to CUDA/hardware specific libraries in general, discussed in the original PR here: #225513 (comment).

Thanks, I didn't know about this issue, but I assumed CUDA wouldn't be working yet.

However, it looks a bit like a straightforward version incompatibility. One thing to try might be to bump the General registry.

Yes, it looks like some straightforward incompatibility. Installing using Pkg inside julia-bin works well. But that of course defeats the purpose of using Nix.

NixOS / nixpkgs