search

dpfried / incoder

Generative model for code infilling and synthesis
291 stars 25 forks source link

Failed to load Half precision 6B model #5

Closed rgzn-aiyun closed 2 years ago

rgzn-aiyun commented 2 years ago

Weird?

return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled) RuntimeError: expected scalar type Float but found Half

dpfried commented 2 years ago

Can you please share a minimal example of code that causes this, and which versions of torch and transformers you are using?

rgzn-aiyun commented 2 years ago

Can you please share a minimal example of code that causes this, and which versions of torch and transformers you are using?

python example_usage.py PyTorch 1.11

Use the cpu to load, is that the case?

dpfried commented 2 years ago

It seems mixed precision, which the .half() / float16 version of the model uses, isn't well-supported on CPU, so the float16 version of the 6B model can only run on GPU.

I've pushed a change that only loads float16 if CUDA == True.

Thanks for catching!