Closed MichaelSp closed 1 year ago
You should be able to add the flag here https://github.com/nat/openplayground/blob/7f3f79035b3229aa471a9746d9af59035d5e71aa/server/lib/inference/huggingface/hf.py#LL38C13-L38C13 and that should allow you to load and infer no problem
Thank you, I can confirm that this worked.
config = AutoConfig.from_pretrained(model_name, trust_remote_code=True) # load config for model
if config.architectures:
model_classname = config.architectures[0]
model_class = getattr(MODULE, model_classname) # get model class from config
model = model_class.from_pretrained(model_name, config=config, trust_remote_code=True) # dynamically load right model class for text generation
else:
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto' if DEVICE == 'cuda' else None, trust_remote_code=True)
now waiting for https://github.com/ggerganov/llama.cpp/issues/1602
mosaicml/mpt-7b-instruct
needs the same extra trust_remote_code=True
as well, so this might be a more general thing?
I tried running the awesome https://huggingface.co/tiiuae/falcon-7b-instruct via Docker and received this error message:
I struggle to find the place where to add the
trust_remote_code=True
. Can you please help me?