Closed kenthua closed 10 months ago
it works with ray-llm (import it instead of aviary), as shown here
I could deploy and test mistralai/Mistral-7B-v0.1
hi @kenthua did the fix described by @mahmedk work for you?
I didn't get a chance to test on T4's like my original post. So maybe it has something to do with that.
With L4s on 0.4.0
I was able to load and serve mistralai/Mistral-7B-v0.1
- this one works.
The same setup with tiiuae/falcon-7b
I get the stacktrace below:
(ServeController pid=1174) engine_args, engine_configs = ray.get(
(ServeController pid=1174) ray.exceptions.RaySystemError: System error: No module named 'transformers_modules'
(ServeController pid=1174) traceback: Traceback (most recent call last):
(ServeController pid=1174) WARNING 2023-11-01 14:31:59,123 controller 1174 application_state.py:663 - The deployments ['VLLMDeployment:tiiuae--falcon-7b'] are UNHEALTHY.
haven't had a chance to test further, but likely needing to add some applications.runtime_env.pip
libraries?
disregard the library reference, but using the meta 7b model as the base, setting trust_remote_code
to false and reducing the tokens to 2048
in both batched and total allowed the model to spin up.
this was tested with 0.4.0
The examples load and serve without issue
meta-llama/Llama-2-7b-chat-hf
andamazon/LightGPT
models.However, anytime I try other models such as
tiiuae/falcon-7b
I end up with the following error trying to load up the following models.