search

dvlab-research / LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)
http://arxiv.org/abs/2309.12307
Apache License 2.0
2.59k stars 267 forks source link

Any chance there will be support for Phi 1.5? #33

Closed jordancole21 closed 10 months ago

jordancole21 commented 11 months ago

Curious if this will work for Microsoft/phi-1_5 or any plans to add support for it since I know it uses the CodeGenTokenizer.

TheSouthFrog commented 11 months ago

We are planning to add support for phi-1.5 model by providing the phi15_attn_replace similar to the gptneox_attn_replace for GPT-NeoX we've just merged, as well as the training script.

However, due to limited computation resources as we're updating the LongQA dataset, finetuned checkpoints might be our first to-dos at this moment. As soon as the LongQA dataset is released (soon), your contribution to training are also welcomed.