search

ray-project / ray-llm

RayLLM - LLMs on Ray
https://aviary.anyscale.com
Apache License 2.0
1.2k stars 87 forks source link

Ray-LLM Head with VLLM Head throws configuration error #114

Open lynkz-matt-psaltis opened 6 months ago

lynkz-matt-psaltis commented 6 months ago

It looks like ray is receiving log_stats from VLLM in the remote call and then amending the config to explicitly add log_stats again?

(ServeController pid=501) RuntimeError: Traceback (most recent call last):
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 443, in initialize_and_get_metadata
(ServeController pid=501)     await self._initialize_replica()
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 182, in initialize_replica
(ServeController pid=501)     await sync_to_async(_callable.__init__)(*init_args, **init_kwargs)
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/rayllm/backend/server/vllm/vllm_deployment.py", line 37, in __init__
(ServeController pid=501)     await self.engine.start()
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/rayllm/backend/llm/vllm/vllm_engine.py", line 81, in start
(ServeController pid=501)     self.engine = self._engine_cls.from_llm_app(
(ServeController pid=501)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/rayllm/backend/llm/vllm/vllm_compatibility.py", line 169, in from_llm_app
(ServeController pid=501)     engine = cls(
(ServeController pid=501)              ^^^^
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 273, in __init__
(ServeController pid=501)     self.engine = self._init_engine(*args, **kwargs)
(ServeController pid=501)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 318, in _init_engine
(ServeController pid=501)     return engine_class(*args, **kwargs)
(ServeController pid=501)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ServeController pid=501)   File "/home/ray/anaconda3/lib/python3.11/site-packages/rayllm/backend/llm/vllm/vllm_compatibility.py", line 37, in __init__
(ServeController pid=501)     super().__init__(*args, **kwargs)
(ServeController pid=501) TypeError: LLMEngine.__init__() got multiple values for argument 'log_stats'
xuhaitao commented 3 months ago

I have encountered the same issue as well, have you resolved it?

lynkz-matt-psaltis commented 3 months ago

Vllm is simply moving too quickly with multiple breaking changes for ray-llm. Given the last significant update was three months ago for rayllm, I'm not sure I can offer you a viable suggestion. Hopefully when the team comes up for a breath we can discuss steps to unblock this. Sorry I couldn't be more help