search

ray-project / ray-llm

RayLLM - LLMs on Ray
https://aviary.anyscale.com
Apache License 2.0
1.22k stars 87 forks source link

Fix serve config parsing #66

Closed gvspraveen closed 11 months ago

gvspraveen commented 11 months ago

This seems to have been accidentally removed in previous PR serve config.yaml expects args to be dictionary. See existing readme in ray-llm repo

Accidentally remove RouterArgs parsing is causing the following issue

pydantic.error_wrappers.ValidationError: 2 validation errors for VLLMApp
engine_config
  field required (type=value_error.missing)
scaling_config
  field required (type=value_error.missing)

Tested locally with following serve config and verified with fix i am able to start server

applications:
- name: meta-llama--Llama-2-7b-chat-hf
  route_prefix: /
  import_path: aviary.backend:router_application
  args:
    models:
      - "path to model"

Looks like others are also facing this: https://ray-distributed.slack.com/archives/C053J39MJ3A/p1696320039604519