python3Packages.torch: "unbundle" git submodules

Describe the bug

Pytorch "vendors" a number of dependencies via git-submodules. The bundled libraries are complex enough that they suffer from the "cabal hell"/the "diamond problem"[^1]. The Nixpkgs' pytorch expression currently keeps the submodules as is, which means that:

...we cannot configure or consistently apply patches to torch's dependencies,
cannot control the bootstrap process,
and potentially suffer the increased build times (as we cannot cache the built dependencies as separate store paths)
and increased closure sizes.

Pytorch's CMakeLists implement a custom mechanism, USE_SYSTEM_LIBS, that partially recovers the ability for dependency injection. Nonetheless, unbundling all of the vendored libraries apparently still takes substantial patching[^1][^2]^3.

The Guix-HPC project has, apparently, made great progress addressing this issue[^1][^2]. Nixpkgs needs to catch up, but also we need to work with the upstream to render patching unnecessary.

[^1]: "What's in a package": https://hpc.guix.info/blog/2021/09/whats-in-a-package/ [^2]: "[PATCH 00/31] Unbundle and update python-pytorch": https://issues.guix.gnu.org/69591 [^3]: "PyTorch with ROCm": https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00242.html

NixOS / nixpkgs

python3Packages.torch: "unbundle" git submodules #239211

Describe the bug