ValueError: model.decoder.embed_tokens.weight doesn't have any device set.

paperswithcode / galai

Model API for GALACTICA

Apache License 2.0

2.68k stars 276 forks source link

ValueError: model.decoder.embed_tokens.weight doesn't have any device set. #40

Closed mwanzu closed 1 year ago

mwanzu commented 1 year ago

i have downloaded the huge model and i get the above error when i run the example below import galai as gal model = gal.load_model("huge", 'float16', 1) model.generate("Scaled dot product attention:\n\n\[", device_map="auto")

mkardas commented 1 year ago

Hi @mwanzu, can you check on galai 1.1.0? Also, with num_gpus=1 you'll get Out of Memory Exception, as the "huge" model weights are 240 GB in dtype=float16.