search

davmacario / MDI-LLM

Implementation of Model-Distributed Inference for Large Language Models, built on top of LitGPT
MIT License
3 stars 2 forks source link

Torch >= 2.2.0 bug on MPS #8

Open davmacario opened 7 months ago

davmacario commented 7 months ago

When running inference from a pretrained model (huggingface) using MPS and a torch version above 2.2.0, I only get '!' at output (token 0).