search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
135.71k stars 27.17k forks source link

Plans to Integrate LongRoPE into LLaMA? #31992

Open ryan-minato opened 4 months ago

ryan-minato commented 4 months ago

Feature request

Microsoft has introduced their microsoft/LongRoPE implementation. Unlike plug-and-play solutions, LongRoPE requires hyperparameter tuning via a genetic algorithm. This implementation is likely the same as described in the Su on Phi-3. Are there any plans to incorporate LongRoPE into LLaMA?

Motivation

In my research on long content, I have managed to integrate LongRoPE into LLaMA with some minor code adjustments. I am curious if Huggingface is also working on integrating this feature.

Your contribution

If necessary.

amyeroberts commented 4 months ago

cc @ArthurZucker @gante

gante commented 4 months ago

Hey @ryan-minato 👋

Thank you for opening this issue! We're pausing new rope scaling contributions for a week or so, while we refactor the code. Past that, we'd love to get a contribution 🤗

ArthurZucker commented 4 months ago

Will be fixed by #31999

gante commented 4 months ago

@ryan-minato #31999 will include longrope 🤗

clemens-magg commented 2 weeks ago

Hi @ryan-minato, is there a chance to make the implementation of your llama-longrope public? :)

ryan-minato commented 2 weeks ago

你好@ryan-minato，是否有机会公开你的 llama-longrope 的实现？:)

Sorry, as I work for a company, I don't hold the copyright for the previous code and therefore can’t share it. I’ll rewrite it from scratch when I have time.