search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
8.26k stars 846 forks source link

Can we integrating LongRoPE. #970

Open PSanni opened 4 months ago

PSanni commented 4 months ago

They have implemented LongRope patch for llama and mistral. Is it possible to port same into ligGPT ?

https://github.com/datamllab/LongLM/tree/master

rasbt commented 4 months ago

Hah, I was just bringing this up to my colleagues yesterday. I think it might be a nice addition in the future.

ryan-minato commented 1 month ago

The Git repo you provided, LongLM, appears to be an implementation of Self-Extend, not LongRoPE. Did I misunderstand something?