Mistral 7B v0.1 does not support optimum BetterTransformers for better and optimized Inference

mistralai / mistral-inference

Official inference library for Mistral models

https://mistral.ai/

Apache License 2.0

9.37k stars 817 forks source link

Mistral 7B v0.1 does not support optimum BetterTransformers for better and optimized Inference #128

Open KaifAhmad1 opened 5 months ago

KaifAhmad1 commented 5 months ago

Raising issue: Facing GPU resource constraints with Mistral-7B-v0.1. Seeking optimizations for VRAM usage and inference performance. Considering alternative solutions due to BetterTransformers not being supported. Open to collaboration on resolving this.