search

nlpxucan / WizardLM

LLMs build upon Evol Insturct: WizardLM, WizardCoder, WizardMath
9.19k stars 713 forks source link

load mush slower than starcoder #76

Closed aliswel-mt closed 1 year ago

aliswel-mt commented 1 year ago

Have someone loaded by AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.float16, device_map="auto) I have tried load starcoder model with A100-40G with approximately 140s to load but it took me 600+ s to load wizardcoder Can someone help ? Is this matter? : starcoder is separated in 7 pieces while wizardcode is a large bin file

ChiYeungLaw commented 1 year ago

I think this is a possible reason. I took about 440s to load wizardcoder from huggingface.

aliswel-mt commented 1 year ago

@ChiYeungLaw Thanks for your response, Did you manage to load the model in a faster way?