tdrussell qlora-pipe issues

tdrussell / qlora-pipe

A pipeline parallel training script for LLMs.

MIT License

79 stars 5 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Training crashes with Accelerate 0.33.0

#22 rationalism opened 1 week ago
0
feature: proportional eval steps

#21 kallewoof opened 2 weeks ago
0
LoRA+ support

#20 kallewoof closed 2 days ago
5
multi-node training error during eval/best_loss checkpoint saving: Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2315, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1808543 milliseconds before timing out.

#19 iamhappytoo opened 1 month ago
2
Softcapping for lm head logits?

#18 kallewoof closed 1 month ago
2
qol: write out a UTF-8 plot on screen showing eval loss history, if plotille is installed

#17 kallewoof closed 1 month ago
3
How to correctly set bf16 for training, and seems deepspeed config additional offload/bf16 conflicts with the qlora-pipe setting?

#16 iamhappytoo opened 2 months ago
4
bug-fix: use right varname in lm_head

#15 kallewoof closed 2 months ago
0
qol: show ETA as a human readable time

#14 kallewoof closed 3 months ago
0
feature: track and hold onto best eval model

#13 kallewoof closed 2 months ago
0
feature: add --resume_from_checkpoint flag

#12 kallewoof closed 3 months ago
0
trivial: allow commented-out resume_from_checkpoint config to mean 'false'

#11 kallewoof closed 3 months ago
0
merge_lora: detect and use safetensors lora when present

#10 kallewoof closed 3 months ago
0
add old-state trimming and handle saving in new module

#9 kallewoof closed 3 months ago
2
improve steps logging including adding an ETA

#8 kallewoof closed 3 months ago
1
fix: avoid empty batches by including incomplete batches when there are no complete ones

#7 kallewoof closed 3 months ago
6
qol: warn user when they are about to spend a lot of time evaluating vs training

#6 kallewoof closed 3 months ago
2
merge_lora: switch copy order and add tqdm

#5 kallewoof closed 3 months ago
0
cleanup: fix imports

#4 kallewoof closed 3 months ago
0
fix: deal gracefully with warmup_steps=0

#3 kallewoof closed 3 months ago
0
DistributedBatchSamper crashes on small eval sample sizes

#2 accupham closed 3 months ago
1
Could you provide an example of a 70B model training config?

#1 accupham opened 5 months ago
4