search

pytorch / torchtitan

A native PyTorch Library for large model training
BSD 3-Clause "New" or "Revised" License
2.57k stars 198 forks source link

add Llama 3.2 support #625

Open tianyu-l opened 2 weeks ago

tianyu-l commented 2 weeks ago 
### Tasks
- [ ] https://github.com/pytorch/torchtitan/pull/589
- [ ] https://github.com/pytorch/torchtitan/pull/626
- [ ] Numerically stable validation for ViT
- [ ] Numerically stable validation for decoder
- [ ] Enable dataloading for dataset OBELICS and update the tokenizer
- [ ] enabling loading the pretrained checkpoints for ViT and non-crossed layer in decoder
- [ ] Training receipe for MM model
- [ ] Perf and numerically stability for 1D (FSDP) for MM model
TJ-Solergibert commented 2 weeks ago

Hi @tianyu-l, is anyone working on the Enable dataloading for dataset OBELICS and update the tokenizer task? Let me know if I could contribute to this task!

tianyu-l commented 2 weeks ago

Hi @TJ-Solergibert, it'd be great if you can help on that. I can help discuss and review the code / design / etc. You may create a separate issue (and add the link above) to track progress if needed.

fduwjj commented 2 weeks ago

@TJ-Solergibert you can add me as the code reviewer as well.