search

dvlab-research / LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)
http://arxiv.org/abs/2309.12307
Apache License 2.0
2.62k stars 274 forks source link

Is it possible to increase the context length of phi-2 using LongLora? If yes, what changes need to be done to support it? #169

Open dbanka opened 9 months ago

dbanka commented 9 months ago

Is it possible to increase the context length of phi-2 using LongLora? If yes, what changes need to be done to support it?

agokrani commented 9 months ago

I am also interested in doing that, I have tried changing the code using LongQLoRA which uses part most of the code of this repository. However, I am getting a strange issue of illegal memory access in cuda. Not really sure why is that the case.