search

bigcode-project / starcoder.cpp

C++ implementation for 💫StarCoder
441 stars 36 forks source link

Inference with Starcoder model finetuned by lora #18

Open edwardelric1202 opened 1 year ago

edwardelric1202 commented 1 year ago

Hi, Can you give some advice about how to inference finetuned Starcoder model with this code? Since lora finetune changed some of layers of the model, some of the code in starcoder.cpp should be changed, how can I use this code to inference with my finetuned Starcoder model?

NouamaneTazi commented 1 year ago

Unfortunately this is not supported yet, but I think llama.cpp does support it. It would be great if somebody can adapt that to starcoder.cpp https://github.com/ggerganov/llama.cpp/pull/820