[SIIT] Convert MPT to support gradient checkpointing, LoRa

[ArthurMinovsky]

Description:

Add MPT with Gradient Checkpointing and LoRa support into OpenThaiGPT pertaining code. We will use MPT with Lora for continue pertaining to task #179

To Do:

1. MPT Weight + MPT with Lora -> Add Lora with PEFT -> Finetune -> Convert to Huggingface checkpoint
2. Implement MPT Lora with OpenThaiGPT GitHub

Outcomes:

• [ ] Jupyter Notebook code that demonstrate that
  • [ ] MPT with Lora can be fine-tuned with PEFT and Huggingface Trainer
  • [ ] MPT with Lora can be convert back to Huggingface checkpoint
• [ ] MPT Lora code that is compatible with the pertaining pipeline with lightning fabric on OpenThaiGPT Github

References:

• MPT with Lora code: https://github.com/iwalton3/mpt-lora-patch

• MPT Weight: https://huggingface.co/mosaicml/mpt-1b-redpajama-200b

• PEFT to implement Lora Finetuning: https://github.com/huggingface/peft

• OpenThaiGPT Github: https://github.com/OpenThaiGPT/openthaigpt-pretraining/tree/main/src/model/openthaigpt_pretraining_model/models

[Wintoplay]

new repository: https://github.com/mikeybellissimo/LoRA-MPT/tree/main

https://colab.research.google.com/drive/1rPwsy659VxOMyx7yUiYo6tsfsg_SyAuf?usp=sharing

still fail: loss == 0

[Chawak]

Resource

• https://github.com/mikeybellissimo/LoRA-MPT/issues/6
• https://colab.research.google.com/drive/1yQZdMdZlMwpniF2cQlTAPnNWqZSDmmb1?usp=sharing

[Wintoplay]

https://colab.research.google.com/drive/1qdRMaJ-ro2k7k23d-ICoBq4IjSwiaTnA?usp=sharing

[Wintoplay]

@pitikorn32 (@boss-chanon บอกให้ tag) repo: https://github.com/mikeybellissimo/LoRA-MPT/tree/main

error message after upgrading peft lib: /tmp/tmpfgiig51b/main.c:2:10: fatal error: cuda.h: No such file or directory

include "cuda.h"