Closed jordancole21 closed 10 months ago
We are planning to add support for phi-1.5 model by providing the phi15_attn_replace
similar to the gptneox_attn_replace
for GPT-NeoX we've just merged, as well as the training script.
However, due to limited computation resources as we're updating the LongQA dataset, finetuned checkpoints might be our first to-dos at this moment. As soon as the LongQA dataset is released (soon), your contribution to training are also welcomed.
Curious if this will work for Microsoft/phi-1_5 or any plans to add support for it since I know it uses the CodeGenTokenizer.