In the Readme it says, mistralai/Mixtral-8x7B-Instruct-v0.1 is supported but Runpod UI doesn't allow more than 1 GPU Worker for 80 GB GPU Cards. Hence, the model can't be served. Is there any other way to serve the original model without using any quantization version?
In the Readme it says,
mistralai/Mixtral-8x7B-Instruct-v0.1
is supported but Runpod UI doesn't allow more than 1 GPU Worker for 80 GB GPU Cards. Hence, the model can't be served. Is there any other way to serve the original model without using any quantization version?