search

dvlab-research / LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)
http://arxiv.org/abs/2309.12307
Apache License 2.0
2.59k stars 267 forks source link

Qustions about dynamic NTK interpolation fine-tuning and non-linear interpolation methods #142

Open Yiyi-philosophy opened 9 months ago

Yiyi-philosophy commented 9 months ago

Hello LongLora Team,

I have been following your work with great interest, particularly regarding the dynamic NTK. While reviewing the code and paper, I noticed that the current experimental results are primarily based on the PI (linear interpolation) method. I am curious to know if you have conducted experiments on interpolation fine-tuning with dynamic NTK. If so, I would be interested in learning more about the corresponding results and findings.

Additionally, I am intrigued by the possibility of experiments with non-linear interpolation methods such as YaRN and NTK. Have you explored similar approaches, and if there are any relevant experiments or findings, I would like to know more about them.

Thank you for your time and efforts!

yukang2017 commented 9 months ago

Hi,

Many thanks for your question. Actually, we have not conducted these experiments yet.

Regards, Yukang Chen