dvlab-research LongLoRA issues

dvlab-research / LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

http://arxiv.org/abs/2309.12307

Apache License 2.0

2.59k stars 267 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

LongBench evaluation

#194 Clement25 opened 1 week ago
0
是否支持如GPT2这类的supervised fine-tune？

#193 CharRic opened 2 weeks ago
0
How LongAlpaca Data was constructed?

#192 S1s-Z opened 1 month ago
0
这套代码是否支持qwen/baichuan微调一个中文的长文本模型，代码需要做哪些修改？

#191 jy-101361-1810897 opened 1 month ago
0
norm层不是没有参数矩阵吗

#190 changanxunyi opened 3 months ago
0
Update README.md

#189 Dominic789654 opened 3 months ago
0
I am unable to reproduce the results from the paper for llama-7B-32k-longlora ppl.

#188 masteryqq opened 3 months ago
1
模型完全没法正常输出

#187 Tangent-90C closed 3 months ago
1
embedding 为什么要resize成32001？

#186 momandai opened 3 months ago
0
Something wrong with the torch version

#185 dian1414 opened 3 months ago
0
What's the trainset is used to obtain “Model with contextg extension via improved LoRA fine-tuning” (LoRA+)？

#184 ZackZikaiXiao opened 4 months ago
0
How did make questions and answers for long context(LongAlpaca)?

#183 ddoyles opened 6 months ago
0
When I set `per_device_train_batch_size=2`, the S2-Attn would not shift as expected

#182 linhaojia13 opened 6 months ago
2
HF models missing rope scaling in the config

#181 hsiehjackson opened 6 months ago
0
Machine don't install Flash Attention

#180 huilong-chen opened 6 months ago
0
global_step文件

#179 xxcoco763 opened 6 months ago
0
Add callback for saving trainable parameters and model config

#178 GirinMan opened 6 months ago
0
Regarding the results in Table 8 and Table 14

#177 Statisticss opened 7 months ago
0
About the different datasets and corresponding models

#176 Statisticss opened 7 months ago
0
The proof-pile/test-sample-ids is not the exact ids for the proof-pile-testsample.bin

#175 pangjh3 closed 7 months ago
0
Memory usage "too small" for 7B Llama-2

#174 Linohong opened 7 months ago
0
training a LLM w/ shifted sparse attention from the scratch?

#173 we1k opened 7 months ago
0
merge_lora_weights_and_save_hf_model.py Error while deserializing header: HeaderTooLarge

#172 Spongeorge opened 7 months ago
0
Distributed inference issue

#171 yixliu1 opened 7 months ago
0
论文中的evaluate结果，推理时用的attention是shifted sparse attention？还是full attention？

#170 zhangxiann opened 7 months ago
0
Is it possible to increase the context length of phi-2 using LongLora? If yes, what changes need to be done to support it?

#169 dbanka opened 7 months ago
1
the value of loss is too unstable when supervised-finetune the 7b-100k-ft model

#168 seanxuu opened 7 months ago
1
streaming llm problem

#167 seanxuu opened 7 months ago
0
How can I use the Llama-2-7b-longlora-100k-ft model correctly

#166 seanxuu opened 7 months ago
0
bug report : RuntimeError: probability tensor contains either inf, nan or element < 0

#165 seanxuu opened 7 months ago
0
Is LongLoRA can be mixed with YaRN ?

#164 DevNullx64 opened 8 months ago
0
推理时候显存分配

#163 xxcoco763 opened 8 months ago
2
Adapting to new models

#162 epinnock opened 8 months ago
2
如何在LoRA训练中加入embed和norm层的训练？

#161 Zheng-Jay opened 8 months ago
0
Lora+deepspeed zero3 无法保存lora权重问题

#160 AresXD closed 8 months ago
6
What llama attn replacement to use for SFT-based inference?

#159 spring1915 opened 8 months ago
0
在没有报错的情况下，LongAlpaca-7B只对文本的第一段文字进行了响应

#158 waleyW opened 8 months ago
0
Configs in inference.py necessary for context length expansion in model serving?

#157 spring1915 opened 8 months ago
0
训练的时候使用的什么外推方式

#156 IT-five opened 8 months ago
0
支持qwen、baichuan等中文模型微调吗

#155 kevinuserdd opened 8 months ago
0
inference OOM

#154 PharMolix opened 9 months ago
0
Is LongAlpaca model fine-tuned from llama-2 or the Alpaca model?

#153 Mooler0410 opened 9 months ago
0
Can LongLoRA be used for incremental pre-training?

#152 Zheng-Jay opened 9 months ago
0
the current text generation call will exceed the model's predefined maximum length (4096)

#151 waleyW opened 9 months ago
4
微调数据

#150 Go4miii closed 9 months ago
0
推理 group整除问题

#149 Michelleable closed 9 months ago
1
LongLoRA + Flash Attention 2 causing illigal memory access

#148 ArturNiederfahrenhorst opened 9 months ago
7
32k inference result is garbled

#147 zhanglv0209 opened 9 months ago
8
torch.cuda.OutOfMemoryError: CUDA out of memory

#146 zhanglv0209 closed 9 months ago
3
中文领域进展

#145 ccp123456789 closed 9 months ago
1