This change adds the --distributed_adam, --distributed_lamb, and --group_norm build options to the version of Apex bundled with torch-extras. Apex is updated to its latest release to add --group_norm support.
This also includes fixes for torch-extras's CI so that it automatically rebuilds itself from the most recent main-branch ml-containers/torch builds when:
The torch-extras/** code itself is changed, and
ml-containers/torch is not also being rebuilt simultaneously.
Previously it could only rebuild on new builds of ml-containers/torch or manual workflow dispatch because we had no way to discover the correct bases to use, or to handle concurrent triggering alongside a ml-containers/torch build.
It will still rebuild from either subset of the torch:base or torch:nccl images as necessary as long as it doesn't overlap with a concurrent run of torch-base.yml or torch-nccl.yml workflows, respectively. Eliminating overlap is necessary to not produce two images with identical tags but different content, which would overwrite each other when published, and leave the actual contents of the image indeterminate.
This also removes the label distinction between our large and standard CI runners in all workflow files, as the distinction no longer exists on the runners themselves, which was blocking torch-extras builds that had been requesting them.
Expand Apex & CI Reliability
This change adds the
--distributed_adam
,--distributed_lamb
, and--group_norm
build options to the version of Apex bundled withtorch-extras
. Apex is updated to its latest release to add--group_norm
support.This also includes fixes for
torch-extras
's CI so that it automatically rebuilds itself from the most recentmain
-branchml-containers/torch
builds when:torch-extras/**
code itself is changed, andml-containers/torch
is not also being rebuilt simultaneously.Previously it could only rebuild on new builds of
ml-containers/torch
or manual workflow dispatch because we had no way to discover the correct bases to use, or to handle concurrent triggering alongside aml-containers/torch
build.It will still rebuild from either subset of the
torch:base
ortorch:nccl
images as necessary as long as it doesn't overlap with a concurrent run oftorch-base.yml
ortorch-nccl.yml
workflows, respectively. Eliminating overlap is necessary to not produce two images with identical tags but different content, which would overwrite each other when published, and leave the actual contents of the image indeterminate.This also removes the label distinction between our large and standard CI runners in all workflow files, as the distinction no longer exists on the runners themselves, which was blocking
torch-extras
builds that had been requesting them.