ml-containers/nightly-torch, versions of ml-containers/torch built every night (05:00 UTC) with the latest HEAD commits from pytorch/pytorch, pytorch/vision, and pytorch/audio
ml-containers/nightly-torch-extras, versions of ml-containers/torch-extras built using ml-containers/nightly-torch as a base instead of ml-containers/torch
Not nightly versions of the extras themselves, but this change additionally updates DeepSpeed to v0.10.1 up from v0.9.4 for better compatibility with newer requirements for PyTorch extensions
DeepSpeed's AIO extension is disabled for builds with torch 2.1.x because it is currently incompatible (see microsoft/DeepSpeed#3976)
These builds are documented after torch and torch-extras in the README index.
Versioning
Published images are tagged with ⟨commit⟩-⟨edition⟩-⟨YYYY.MM.DD.HH⟩-⟨version info⟩ to prevent tag conflicts when repeatedly building from the same commit
Library version numbers are determined by the /version.txt files in each pytorch/* repo, since there aren't Git tags to use
This retains important information about a commit source while also remaining compatible with requirements.txt specifications such as torch>=2.0.0
Structural Changes
.github/workflows/torch-base.yml now pulls its configuration information from .github/configurations/torch-base.yml via the reusable workflow .github/workflows/read-configuration.yml
This includes a list of base images, plus library versions
.github/workflows/torch-nightly.ymlpulls from both, but supplies its own library versions
This was originally set up so that torch-nightly.yml would call torch-base.yml and torch-nccl.yml as reusable workflows to inherit their configurations, but that ran up against the GitHub Actions reusable workflow call depth limit of three due to the chain torch-nightly.yml > torch-____.yml > torch.yml > build.yml. This setup flattens the call chain enough to not hit the limit, but to still share a source of configuration information with the two other workflows to stay in sync.
Build Process Improvements
Compilers have been updated to gcc-11 and g++-11 up from gcc-10 and g++-10
gcc-11 and g++-11 are the first GCC versions that default to compiling with C++17, and torch 2.1.x requires compilation with C++17 for all extensions
The linker has been updated to lld-17
torch:
Can now build from commit hashes in addition to Git tags
Can now build with custom versions of openai/triton
Uses an increased ccache size
Employs better build multiprocessing via effective_cpu_count.sh
Nightly PyTorch Builds
This change adds experimental container builds:
ml-containers/nightly-torch
, versions ofml-containers/torch
built every night (05:00 UTC) with the latest HEAD commits from pytorch/pytorch, pytorch/vision, and pytorch/audioml-containers/nightly-torch-extras
, versions ofml-containers/torch-extras
built usingml-containers/nightly-torch
as a base instead ofml-containers/torch
These builds are documented after
torch
andtorch-extras
in the README index.Versioning
⟨commit⟩-⟨edition⟩-⟨YYYY.MM.DD.HH⟩-⟨version info⟩
to prevent tag conflicts when repeatedly building from the same commit/version.txt
files in eachpytorch/*
repo, since there aren't Git tags to usetorch==2.1.0a0+78810d7
requirements.txt
specifications such astorch>=2.0.0
Structural Changes
.github/workflows/torch-base.yml
now pulls its configuration information from.github/configurations/torch-base.yml
via the reusable workflow.github/workflows/read-configuration.yml
.github/workflows/torch-nccl.yml
pulls from.github/configurations/torch-nccl.yml
.github/workflows/torch-nightly.yml
pulls from both, but supplies its own library versionsThis was originally set up so that
torch-nightly.yml
would calltorch-base.yml
andtorch-nccl.yml
as reusable workflows to inherit their configurations, but that ran up against the GitHub Actions reusable workflow call depth limit of three due to the chaintorch-nightly.yml > torch-____.yml > torch.yml > build.yml
. This setup flattens the call chain enough to not hit the limit, but to still share a source of configuration information with the two other workflows to stay in sync.Build Process Improvements
torch
:openai/triton
ccache
sizeeffective_cpu_count.sh
Published Images
Some container images are already published to ml-containers/pkgs/container/ml-containers%2Fnightly-torch. Some
nightly-torch-extras
container images were published under ml-containers/pkgs/container/ml-containers%2Ftorch-extras instead of their own container name because of a bug that has been fixed, and new builds are set to use the namenightly-torch-extras
. Thecron
schedule to build all of these will start once this is merged tomain
.