shibing624 / textgen

TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型,实现了包括LLaMA,ChatGLM,BLOOM,GPT2,Seq2Seq,BART,T5,UDA等模型的训练和预测,开箱即用。
Apache License 2.0
937 stars 109 forks source link

如何用使用 p40 训练lora #21

Closed hongyix closed 1 year ago

hongyix commented 1 year ago

Describe the bug

torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised RuntimeError: Found Tesla P40 which is too old to be supported by the triton GPU compiler, which is used as the backend. Triton only supports devices of CUDA Capability >= 7.0, but your device is of CUDA capability 6.1

To Reproduce

image

Describe your attempts

按照https://www.reddit.com/r/StableDiffusion/comments/12nh9hn/dreambooth_training_lora_failed_due_to_triton_gpu/ 上说的配置export TORCHDYNAMO_DISABLE=1 重跑后会报

RuntimeError: GET was unable to find an engine to execute this computation

Context

shibing624 commented 1 year ago

升级cuda

hongyix commented 1 year ago

升级cuda和最新版本的peft就可以了。