feat(torch): Nightly PyTorch Builds

[Eta0]

Nightly PyTorch Builds

This change adds experimental container builds:

• ml-containers/nightly-torch, versions of ml-containers/torch built every night (05:00 UTC) with the latest HEAD commits from pytorch/pytorch, pytorch/vision, and pytorch/audio
  • Builds with corresponding custom Triton versions pinned in the PyTorch repo
• ml-containers/nightly-torch-extras, versions of ml-containers/torch-extras built using ml-containers/nightly-torch as a base instead of ml-containers/torch
  • Not nightly versions of the extras themselves, but this change additionally updates DeepSpeed to v0.10.1 up from v0.9.4 for better compatibility with newer requirements for PyTorch extensions
  • DeepSpeed's AIO extension is disabled for builds with torch 2.1.x because it is currently incompatible (see microsoft/DeepSpeed#3976)

These builds are documented after torch and torch-extras in the README index.

Versioning

• Published images are tagged with ⟨commit⟩-⟨edition⟩-⟨YYYY.MM.DD.HH⟩-⟨version info⟩ to prevent tag conflicts when repeatedly building from the same commit
• Library version numbers are determined by the /version.txt files in each pytorch/* repo, since there aren't Git tags to use
• The packages installed in the container use those official version numbers along with a local version identifier to store the Git commit hash from which the library was built, as prescribed in PEP 440 § DVCS based version labels
  • E.g. torch==2.1.0a0+78810d7
  • This retains important information about a commit source while also remaining compatible with requirements.txt specifications such as torch>=2.0.0

Structural Changes

• .github/workflows/torch-base.yml now pulls its configuration information from .github/configurations/torch-base.yml via the reusable workflow .github/workflows/read-configuration.yml
  • This includes a list of base images, plus library versions
• .github/workflows/torch-nccl.yml pulls from .github/configurations/torch-nccl.yml
• .github/workflows/torch-nightly.yml pulls from both, but supplies its own library versions

This was originally set up so that torch-nightly.yml would call torch-base.yml and torch-nccl.yml as reusable workflows to inherit their configurations, but that ran up against the GitHub Actions reusable workflow call depth limit of three due to the chain torch-nightly.yml > torch-____.yml > torch.yml > build.yml. This setup flattens the call chain enough to not hit the limit, but to still share a source of configuration information with the two other workflows to stay in sync.

Build Process Improvements

• Compilers have been updated to gcc-11 and g++-11 up from gcc-10 and g++-10
  • gcc-11 and g++-11 are the first GCC versions that default to compiling with C++17, and torch 2.1.x requires compilation with C++17 for all extensions
• The linker has been updated to lld-17
• torch:
  • Can now build from commit hashes in addition to Git tags
  • Can now build with custom versions of openai/triton
  • Uses an increased ccache size
  • Employs better build multiprocessing via effective_cpu_count.sh

Published Images

Some container images are already published to ml-containers/pkgs/container/ml-containers%2Fnightly-torch. Some nightly-torch-extras container images were published under ml-containers/pkgs/container/ml-containers%2Ftorch-extras instead of their own container name because of a bug that has been fixed, and new builds are set to use the name nightly-torch-extras. The cron schedule to build all of these will start once this is merged to main.