Joe/mixtral vllm - Githubissues

replicate / cog-llama-template

LLaMA Cog template

Apache License 2.0

307 stars 52 forks source link

Open joehoover opened 11 months ago

joehoover commented 11 months ago

This PR introduces the following changes:

Add support for mixtral-8x7b-instruct-v0.1.
Update vLLMEngine to except **kwargs to allow overrides
Modify vLLMEngine's delete_lora() method so that vLLM versions that do not implement a delete_lora() method do not fail during setup. This is a bit of a bodge as we currently expect to use our vLLM fork with lora support.

technillogue commented 11 months ago

I thought we weren't checking in tokenizers anymore?