Open rgbayrak opened 9 months ago
Maybe this >> If you loaded your model on the CPU, make sure to move it to a GPU device first. Use the device_map parameter to specify where to place the model:
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="cuda:0")
Make sure the models run on GPU