Modalities / modalities

Modalities, a PyTorch-native framework for distributed and reproducible foundation model training.

MIT License

61 stars 5 forks source link

Fsdp 2.0 integration #206

Open le1nux opened 2 months ago

le1nux commented 2 months ago

What does this PR do?

This PR ..

General Changes

..

Breaking Changes

..

TODOs

[ ] Need to check if we want to use tied weights for the embedding and LM head and how this would be parallelized with FSDP and tensor parallelism

Checklist before submitting final PR

[ ] My PR is minimal and addresses one issue in isolation
[ ] I have merged the latest version of the target branch into this feature branch
[ ] I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
[ ] I have run a sample config for model training
[ ] I have checked that all tests run through (python tests/tests.py)