Closed datquocnguyen closed 6 months ago
model_8bit = AutoModelForCausalLM.from_pretrained("vinai/PhoGPT-4B-Chat", device_map="auto", load_in_8bit=True, trust_remote_code=True)
Or:
config = AutoConfig.from_pretrained("vinai/PhoGPT-4B-Chat", trust_remote_code=True)
config.init_device = "cuda"
model_8bit = AutoModelForCausalLM.from_pretrained("vinai/PhoGPT-4B-Chat", config=config, load_in_8bit=True)
See: https://huggingface.co/docs/transformers/main/en/quantization#bitsandbytes
Or: