search

shibing624 / CodeAssist

CodeAssist is an advanced code completion tool that provides high-quality code completions for Python, Java, C++ and so on. CodeAssist 是一个高级代码补全工具，高质量为 Python、Java 和 C++ 补全代码。
Apache License 2.0
55 stars 9 forks source link

QLoRA and DDP #4

Open mrT23 opened 1 year ago

mrT23 commented 1 year ago

Thanks for the great repo

i have two questions about training the models (specifically WizardCoder):

  1. have you tried training with QLoRa, and not just LoRa ? are you considering adding it to the repo ?

  2. the example usage (https://github.com/shibing624/CodeAssist#train-wizardcoder-model) is without ddp, only dp. are you sure this is the optimal setting? We got significantly higher training rates with ddp

shibing624 commented 1 year ago
  1. qlora will add, next week. qlora done in my repo: https://github.com/shibing624/MedicalGPT
  2. use torchrun
mrT23 commented 1 year ago

The base example is bad (no ddp)

cd examples
CUDA_VISIBLE_DEVICES=0,1 python training_wizardcoder_mydata.py --do_train --do_predict --num_epochs 1 --output_dir outputs-wizard --model_name WizardLM/WizardCoder-15B-V1.0

and there are some discrepancies in the code, that need adjustments to ddp, for example doing 'do_predict' just on the main process.

anyway, thanks for the answer, and the great code.

stale[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.(由于长期不活动，机器人自动关闭此问题，如果需要欢迎提问)