Hydra Release 2021.11 - Githubissues

knagrecha / hydra

Execution framework for multi-task model parallelism. Enables the training of arbitrarily large models with a single GPU, with linear speedups for multi-gpu multi-task execution.

Apache License 2.0

20 stars 3 forks source link

Hydra Release 2021.11 #1

Open knagrecha opened 2 years ago

knagrecha commented 2 years ago

Current Release:

Standard model-parallel sharding supported
Pilot-run style partitioning
Sharded-LRTF Scheduling
Standard linear execution patterns for forward/backward passes
Arbitrarily deep models can be trained on one GPU
Near-linear speedups for end-to-end runtimes in single-node multi-GPU setting

Next Release TODOs (Target Completion: 2022.06):

[ ] Tensor Parallel Support
[ ] Recurrent/Autoregressive Network Support
[ ] Cluster setting scheduler (prioritize turnaround time vs makespan)
[ ] Disk spilling
[ ] Multi-node scaling
[ ] More examples
[ ] Additional documentation
[ ] More robust partitioner

Long-term TODOs

[ ] (Potential) FlexFlow integration?
[ ] Pip package
[ ] AMD GPU support
[ ] Unit tests
[ ] Website
[ ] TensorFlow support
[ ] Data Parallel Support
[ ] Model Selection APIs

knagrecha commented 2 years ago

Also, really need to get support for PyTorch 1.10 and the corresponding torchtext. The codebase itself should be fine, even have some possible optimization opportunities (see saved_intermediates branch) but the examples just break entirely when I try to upgrade PyTorch.