If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.
All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.
Description
HF PT tests failing in release pipeline due to ModuleNotFound for smprocessing.modelparallel (SMMP). After syncing with the SMMP team, I learned the following:
SMP v1 is supported only on PyTorch <2.0 and CUDA 11.
SMP v2 will work on PyTorch 2.1 and CUDA 12, but this version of SMP is used specifically on SMP distributed containers (not DLC), and therefore do not require coverage in this repo.
Despite a lengthly investigation, I cannot confirm why these tests were passing 5 months ago.
HF PT tests for smprocessing.dataparallelprocessing were also failing with RuntimeError: SMDDP does not support: _allgather_base in the script. Following an investigation and a meeting with the SMDDB oncall, I was able to rootcause this and update the test.
Switching to torch_distribution from sm_distribution. These are both libraries providing the building blocks for distributed training. torch_distribution is essentially more generic while sm_distribution is tailored for SageMaker aplpications. torch_distribution however, supports all_gather while sm_distribution does not.
Tests run
NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"
[ ] I have run builds/tests on commit for my changes.
Confused on how to run tests? Try using the helper utility...
Assuming your remote is called `origin` (you can find out more with `git remote -v`)...
- Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:
`python src/prepare_dlc_dev_environment.py -b -cp origin`
- Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example:
`python src/prepare_dlc_dev_environment.py -b -t sanity_tests -cp origin`
- Restore TOML file when ready to merge
`python src/prepare_dlc_dev_environment.py -rcp origin`
NOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:
Expand
- [ ] `sagemaker_remote_tests = true`
- [ ] `sagemaker_efa_tests = true`
- [ ] `sagemaker_rc_tests = true`
**Additionally, please run the sagemaker local tests in at least one revision:**
- [ ] `sagemaker_local_tests = true`
Expand
Fill out the template and click the checkbox of the builds you'd like to execute
*Note: Replace with with the major.minor framework version (i.e. 2.2) you would like to start.*
- [ ] build_pytorch_training__sm
- [ ] build_pytorch_training__ec2
- [ ] build_pytorch_inference__sm
- [ ] build_pytorch_inference__ec2
- [ ] build_pytorch_inference__graviton
- [ ] build_tensorflow_training__sm
- [ ] build_tensorflow_training__ec2
- [ ] build_tensorflow_inference__sm
- [ ] build_tensorflow_inference__ec2
- [ ] build_tensorflow_inference__graviton
Additional context
PR Checklist
Expand
- [ ] I've prepended PR tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron/graviton] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker]
- [ ] If the PR changes affects SM test, I've modified dlc_developer_config.toml in my PR branch by setting sagemaker_tests = true and efa_tests = true
- [ ] If this PR changes existing code, the change fully backward compatible with pre-existing code. (Non backward-compatible changes need special approval.)
- [ ] (If applicable) I've documented below the DLC image/dockerfile this relates to
- [ ] (If applicable) I've documented below the tests I've run on the DLC image
- [ ] (If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list. See [https://www.apache.org/legal/resolved.html](https://www.apache.org/legal/resolved.html).
- [ ] (If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them.
#### NEURON/GRAVITON Testing Checklist
* When creating a PR:
- [ ] I've modified `dlc_developer_config.toml` in my PR branch by setting `neuron_mode = true` or `graviton_mode = true`
#### Benchmark Testing Checklist
* When creating a PR:
- [ ] I've modified `dlc_developer_config.toml` in my PR branch by setting `ec2_benchmark_tests = true` or `sagemaker_benchmark_tests = true`
Pytest Marker Checklist
Expand
- [ ] (If applicable) I have added the marker `@pytest.mark.model("")` to the new tests which I have added, to specify the Deep Learning model that is used in the test (use `"N/A"` if the test doesn't use a model)
- [ ] (If applicable) I have added the marker `@pytest.mark.integration("")` to the new tests which I have added, to specify the feature that will be tested
- [ ] (If applicable) I have added the marker `@pytest.mark.multinode()` to the new tests which I have added, to specify the number of nodes used on a multi-node test
- [ ] (If applicable) I have added the marker `@pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">)` to the new tests which I have added, if a test is specifically applicable to only one processor type
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
GitHub Issue #, if available:
Note:
If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.
All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.
Description
RuntimeError: SMDDP does not support: _allgather_basein the script. Following an investigation and a meeting with the SMDDB oncall, I was able to rootcause this and update the test.torch_distributionfromsm_distribution. These are both libraries providing the building blocks for distributed training.torch_distributionis essentially more generic whilesm_distributionis tailored for SageMaker aplpications.torch_distributionhowever, supportsall_gatherwhilesm_distributiondoes not.Tests run
NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"
Confused on how to run tests? Try using the helper utility...
Assuming your remote is called `origin` (you can find out more with `git remote -v`)... - Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example: `python src/prepare_dlc_dev_environment.py -b -cp origin` - Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example: `python src/prepare_dlc_dev_environment.py -b -t sanity_tests -cp origin` - Restore TOML file when ready to merge `python src/prepare_dlc_dev_environment.py -rcp origin`NOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:
Expand
- [ ] `sagemaker_remote_tests = true` - [ ] `sagemaker_efa_tests = true` - [ ] `sagemaker_rc_tests = true` **Additionally, please run the sagemaker local tests in at least one revision:** - [ ] `sagemaker_local_tests = true`Formatting
black -l 100on my code (formatting tool: https://black.readthedocs.io/en/stable/getting_started.html)DLC image/dockerfile
Builds to Execute
Expand
Fill out the template and click the checkbox of the builds you'd like to execute *Note: Replace withAdditional context
PR Checklist
Expand
- [ ] I've prepended PR tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron/graviton] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker] - [ ] If the PR changes affects SM test, I've modified dlc_developer_config.toml in my PR branch by setting sagemaker_tests = true and efa_tests = true - [ ] If this PR changes existing code, the change fully backward compatible with pre-existing code. (Non backward-compatible changes need special approval.) - [ ] (If applicable) I've documented below the DLC image/dockerfile this relates to - [ ] (If applicable) I've documented below the tests I've run on the DLC image - [ ] (If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list. See [https://www.apache.org/legal/resolved.html](https://www.apache.org/legal/resolved.html). - [ ] (If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them. #### NEURON/GRAVITON Testing Checklist * When creating a PR: - [ ] I've modified `dlc_developer_config.toml` in my PR branch by setting `neuron_mode = true` or `graviton_mode = true` #### Benchmark Testing Checklist * When creating a PR: - [ ] I've modified `dlc_developer_config.toml` in my PR branch by setting `ec2_benchmark_tests = true` or `sagemaker_benchmark_tests = true`Pytest Marker Checklist
Expand
- [ ] (If applicable) I have added the marker `@pytest.mark.model("By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.