pytorch / PiPPy

Pipeline Parallelism for PyTorch
BSD 3-Clause "New" or "Revised" License
726 stars 86 forks source link

Support for Autoregressive generation with LLMs #1136

Open apresunreve opened 3 months ago

apresunreve commented 3 months ago

Hi, I was able to run llama for a single forward, but when I tried to make it generate texts in an autoregressive way, there were errors with input shapes. Looks like a fixed sized input is set when exporting the model (correct me if I'm wrong). I would like to know if LLM text generation is currently supported. Thanks.