feat: Added training scripts Independent of Hugging Face's `Trainer` - Githubissues

turingmotors / heron

Apache License 2.0

165 stars 25 forks source link

feat: Added training scripts Independent of Hugging Face's `Trainer` #27

Open chizuchizu opened 1 year ago

chizuchizu commented 1 year ago

Added training scripts Independent of Hugging Face's `Trainer`

Description

Purpose of the PR

Added training scripts that don't depend on Hugging Face's Trainer class.

Related Issue(s)

Nothing.

Changes

Added train_ds.py
Added utils scripts imported from Microsoft VisualChat.
Added the training config and the training shell script.
Update documentation on training and inference methods.

Model/Algorithm Performance

Trained the GIT model following projects/opt/exp002_ds.yml with both ZeRO-2 and ZeRO-3, confirming valid inference.
Trained Video BLIP with ZeRO-3.
Observed an increase in GPU memory consumption compared to when using Trainer.

Dependencies

Nothing.

Reviewer Notes

Confirmed

[x] I have updated the documentation accordingly.
[x] I have adhered to the coding standards and guidelines of this project.
[x] I have added comments, especially in hard-to-understand areas.