Open Fahmie23 opened 5 months ago
irexyc
commented
5 months ago How did you install LMDeploy? The version 0.4.2 doesn't support openbmb/MiniCPM-Llama3-V-2_5, you should build the latest code or copy the latest code to the installation folder.
Fahmie23
commented
5 months ago How did you install LMDeploy? The version 0.4.2 doesn't support
openbmb/MiniCPM-Llama3-V-2_5, you should build the latest code or copy the latest code to the installation folder.
i install using pip install lmdeploy.
Fahmie23
commented
5 months ago which version do i need to install. I thought version 0.4.2 is the latest version
Fahmie23
commented
5 months ago CancelledError Traceback (most recent call last) File /usr/lib/python3.10/asyncio/tasks.py:234, in Task.__step(failed resolving arguments) 233 else: --> 234 result = coro.throw(exc) 235 except StopIteration as exc:
File /usr/lib/python3.10/asyncio/queues.py:159, in Queue.get(self) 158 try: --> 159 await getter 160 except:
File /usr/lib/python3.10/asyncio/futures.py:285, in Future.await(self) 284 self._asyncio_future_blocking = True --> 285 yield self # This tells Task to wait for completion. 286 if not self.done():
File /usr/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future) 303 try: --> 304 future.result() 305 except BaseException as exc: 306 # This may also be a cancellation.
File /usr/lib/python3.10/asyncio/futures.py:196, in Future.result(self) 195 exc = self._make_cancelled_error() ... --> 178 exit(1) 179 continue 180 except Exception as e:
NameError: name 'exit' is not defined
irexyc
commented
5 months ago 0.4.2 is the latest version published, but openbmb/MiniCPM-Llama3-V-2_5 is added after that release.
To use openbmb/MiniCPM-Llama3-V-2_5, you could use one of the following method.
Fahmie23
commented
5 months ago i already git clone the repo. After that i got new error. Below is the error:
2024-06-13 07:19:06,357 - lmdeploy - ERROR - Engine loop failed with error: MiniCPMV.forward() missing 1 required positional argument: 'data' Traceback (most recent call last): File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/request.py", line 17, in _raise_exception_on_finish task.result() File "/usr/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception.with_traceback(self._exception_tb) File "/usr/lib/python3.10/asyncio/tasks.py", line 232, in step result = coro.send(None) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 824, in async_loop await step(True) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 810, in step raise e File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 802, in step raise out File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 751, in _async_loop_background await self._async_step_background( File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 659, in _async_step_background output = await self._async_model_forward(inputs, File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/utils.py", line 250, in tmp return (await func(*args, **kwargs)) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 576, in _async_model_forward return await forward(inputs) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 554, in forward return await self.model_agent.async_forward( File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 735, in async_forward ... return self._call_impl(*args, *kwargs) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, **kwargs) TypeError: MiniCPMV.forward() missing 1 required positional argument: 'data' Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings... /home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/pygments/regexopt.py:77: RuntimeWarning: coroutine 'AsyncEngine.batch_infer..gather' was never awaited '|'.join(regex_opt_inner(list(group[1]), '') RuntimeWarning: Enable tracemalloc to get the object allocation traceback CancelledError Traceback (most recent call last) File /usr/lib/python3.10/asyncio/tasks.py:234, in Task.step(failed resolving arguments) 233 else: --> 234 result = coro.throw(exc) 235 except StopIteration as exc:
File /usr/lib/python3.10/asyncio/queues.py:159, in Queue.get(self) 158 try: --> 159 await getter 160 except:
File /usr/lib/python3.10/asyncio/futures.py:285, in Future.await(self) 284 self._asyncio_future_blocking = True --> 285 yield self # This tells Task to wait for completion. 286 if not self.done():
File /usr/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future) 303 try: --> 304 future.result() 305 except BaseException as exc: 306 # This may also be a cancellation.
File /usr/lib/python3.10/asyncio/futures.py:196, in Future.result(self) 195 exc = self._make_cancelled_error() ... --> 178 exit(1) 179 continue 180 except Exception as e:
NameError: name 'exit' is not defined
irexyc
commented
5 months ago Clone repo is not enougth.
You should either build LMDeploy yourself (the first two methods) or copy the latest code to the installation folder (/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy).
Fahmie23
commented
5 months ago i already follow your step. After that i got new error. Can you help me. Below is the error:
2024-06-13 07:19:06,357 - lmdeploy - ERROR - Engine loop failed with error: MiniCPMV.forward() missing 1 required positional argument: 'data' Traceback (most recent call last): File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/request.py", line 17, in _raise_exception_on_finish task.result() File "/usr/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception.with_traceback(self._exception_tb) File "/usr/lib/python3.10/asyncio/tasks.py", line 232, in step result = coro.send(None) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 824, in async_loop await step(True) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 810, in step raise e File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 802, in step raise out File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 751, in _async_loop_background await self._async_step_background( File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 659, in _async_step_background output = await self._async_model_forward(inputs, File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/utils.py", line 250, in tmp return (await func(*args, **kwargs)) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 576, in _async_model_forward return await forward(inputs) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 554, in forward return await self.model_agent.async_forward( File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 735, in async_forward ... return self._call_impl(*args, *kwargs) File "/home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, **kwargs) TypeError: MiniCPMV.forward() missing 1 required positional argument: 'data' Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings... /home/mcmcssuserdata/fahmie/lm_deploy/lm_deploy/lib/python3.10/site-packages/pygments/regexopt.py:77: RuntimeWarning: coroutine 'AsyncEngine.batch_infer..gather' was never awaited '|'.join(regex_opt_inner(list(group[1]), '') RuntimeWarning: Enable tracemalloc to get the object allocation traceback CancelledError Traceback (most recent call last) File /usr/lib/python3.10/asyncio/tasks.py:234, in Task.step(failed resolving arguments) 233 else: --> 234 result = coro.throw(exc) 235 except StopIteration as exc:
File /usr/lib/python3.10/asyncio/queues.py:159, in Queue.get(self) 158 try: --> 159 await getter 160 except:
File /usr/lib/python3.10/asyncio/futures.py:285, in Future.await(self) 284 self._asyncio_future_blocking = True --> 285 yield self # This tells Task to wait for completion. 286 if not self.done():
File /usr/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future) 303 try: --> 304 future.result() 305 except BaseException as exc: 306 # This may also be a cancellation.
File /usr/lib/python3.10/asyncio/futures.py:196, in Future.result(self) 195 exc = self._make_cancelled_error() ... --> 178 exit(1) 179 continue 180 except Exception as e:
NameError: name 'exit' is not defined
QwertyJack
commented
5 months ago Seems that you are using the pytorch backend. However, with the latest code from the main branch, pipeline can select the correct backend:
>>> from lmdeploy.turbomind.supported_models import is_supported
>>> is_supported('/data/models/MiniCPM-Llama3-V-2_5')
Truepipe = pipeline('openbmb/MiniCPM-Llama3-V-2_5')
@Fahmie23
If you replaced all files, the above code will use turobmind backend to init the pipeline, but according to your log, there are lines like lmdeploy/pytorch/engine/engine.py saying that you are actually using pytorch backend.
Fahmie23
commented
5 months ago can i know how can i get the latest code?
irexyc
commented
5 months ago Git clone will clone the latest code.
But I have no way to know whether your correctly replaced the installation files. The log information is not right as pytorch backend should not be used.
So I suggest you to build LMDeploy with github action and install the built wheel package.
AnyangAngus
commented
5 months ago @irexyc I run the MINICPMV-2,5 infer in lmdeploy successfully, Nice Work! one question is that the model forward in lmdeploy is based on Fastertransfomer or pytorch ?
since there is a bug in the batch infer in Huggingface modeling_minicpmv.py [https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/discussions/29], so dose lmdeploy supports a correct batch infer? Thank U! ^_^
irexyc
commented
5 months ago @AnyangAngus
Currently, the LLM part of INICPMV-2.5 only support turbomind backend which is developed based on Fastertransfomer. But in the last few versions, almost all kernels except for sampling have been rewritten. And we support batch inference.
AnyangAngus
commented
5 months ago @AnyangAngus
Currently, the LLM part of INICPMV-2.5 only support
turbomind backendwhich is developed based on Fastertransfomer. But in the last few versions, almost all kernels except for sampling have been rewritten. And we support batch inference.
@irexyc Cool!
Another question is that how can i pass the infer parameters to lmdeploy in Minicpmv-2.5 according to its huggingface chat function such as temperature and sampling parameters .
since the text prompt in HF convert to a msgs, dose the lmdeploy minicpm2.5 follow this or I should make a msgs out of the lmdeploy pipe func?
question = 'What is in the image?'
msgs = [{'role': 'user', 'content': question}]
res = model.chat(
image=image,
msgs=msgs,
tokenizer=tokenizer,
sampling=True, # if sampling=False, beam_search will be used by default
temperature=0.7,
# system_prompt='' # pass system_prompt if needed
)
print(res)@AnyangAngus
For offline usage(pipeline api), you can refer to this docs For servering, we provide openai-compatible api usage, you can refer to this docs
@irexyc
As for minicpm offline usage pipeline api, I observe that the backend_config and VisionConfig do not go into effect I reduce the cache_max_entry_count but the GPU memory still need 55G as the same as cache_max_entry_count=0.8 backend_config = TurbomindEngineConfig(cache_max_entry_count=0.2) pipe = pipeline('/models/MiniCPM-Llama3-V-2_5', backend_config=backend_config))
I set the batch size try to reduce the GPU memory, but I still can infer batch items and the GPU memory dose not change. vision_config=VisionConfig(max_batch_size=1) pipe = pipeline('/models/MiniCPM-Llama3-V-2_5', vision_config=vision_config)
Is the way I passed the parameters incorrect？ and How can I pass this parameters to Minicpm2,5 correctly? Many Thanks
QwertyJack
commented
5 months ago @AnyangAngus This issue has been identified and has been resolved through fix #1778.
WANGSSSSSSS
commented
5 months ago which branch should i use for suppoorting the minicpm-v-2.5?
lvhan028
commented
5 months ago the latest main
Checklist
Describe the bug
I try loading and make inference using the sample code given in the github examples directory. The model that i want to load is MiniCPM-Llama3-V-2_5. Could someone help me.
Reproduction
from lmdeploy import pipeline from lmdeploy.vl import load_image
pipe = pipeline('openbmb/MiniCPM-Llama3-V-2_5')
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg') response = pipe(('describe this image', image)) print(response)
Environment
Error traceback