depoyment of quantized models fails

hi,

I try to deploy quantized default models:

serve deploy ray-llm/serve_configs/squeeze-ai-lab--sq-llama-2-7b-w4-s0.yaml

which fails (visible in dashboard):

Deploying app 'ray-llm' failed with exception:
Traceback (most recent call last):
  File "pydantic/main.py", line 522, in pydantic.main.BaseModel.parse_obj
ValueError: dictionary update sequence element #0 has length 1; 2 is required

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/server/utils.py", line 56, in parse_args
    parsed_models = [llm_app_cls.parse_yaml(raw_model)]
  File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/server/models.py", line 59, in parse_yaml
    return cls.parse_obj(dict_args)
  File "pydantic/main.py", line 525, in pydantic.main.BaseModel.parse_obj
pydantic.error_wrappers.ValidationError: 1 validation error for VLLMApp
__root__
  VLLMApp expected dict not str (type=type_error)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/serve/_private/application_state.py", line 909, in build_serve_application
    app = call_app_builder_with_args_if_necessary(import_attr(import_path), args)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/serve/_private/api.py", line 377, in call_app_builder_with_args_if_necessary
    app = builder(args)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/server/run.py", line 115, in router_application
    llm_apps = parse_args(router_args.models, llm_app_cls=VLLMApp)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/server/utils.py", line 59, in parse_args
    raise ValueError(
ValueError: Could not parse string as yaml. If you are specifying a path, make sure it exists and can be reached.

Any ideas?

ray-project / ray-llm

depoyment of quantized models fails #103