DNGros / lmwrapper

An object-oriented wrapper around language models (like openai endpoints or huggingface)
1 stars 1 forks source link

Accelerate Support #23

Open claudiosv opened 11 months ago

claudiosv commented 11 months ago

Accelerate allows training/inference of large models by automagically splitting the layers across CUDA devices. Initially, we had some issues with logprobs due to model forward patching. Now that we generally use transition scores, it may work with accelerate.