🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
This PR introduces the TorchDynamo integration with PyTorch XLA. Though I haven't thorougly benchmarked the code, it works on the CLM example from the Transformers library, when spawned using the accelerate launch command on a v5e TPU VM with 8 TPUs running on Ubuntu 22.04. It also passes all tests I ran on the TPU.
Fixes #2870
Before submitting
[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What does this PR do?
This PR introduces the TorchDynamo integration with PyTorch XLA. Though I haven't thorougly benchmarked the code, it works on the CLM example from the Transformers library, when spawned using the
accelerate launch
command on a v5e TPU VM with 8 TPUs running on Ubuntu 22.04. It also passes all tests I ran on the TPU.Fixes #2870
Before submitting
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
@SunMarc