Closed zysno1 closed 11 months ago
UranusSeven
commented
11 months ago Hi, thanks for reporting this issue!
The error message Not found on the webpage indicates there's something wrong when loading the model. I see you have attached part of the server side logs, could you attach the full logs?
UranusSeven
commented
11 months ago The problem seems to be relavent to XINFERENCE_MODEL_SRC=xorbits. This env var force xinference downloading models from our s3 bucket, which is not accessible right now.
We've added support of downloading from modelscope, and you can enable this feature byXINFERENCE_MODEL_SRC=modelscope.
Bojun-Feng
commented
11 months ago I also encountered a similar issue, Python version is 3.11.4 and using newest version of Xinference on GitHub
I would recommend adding the argument --log-level DEBUG to see what is going on when launching the model, in my case the issue originates from launching the model in the Supervisor before building the gradio page.
The core issue seems to be in the launch model part of the code:
File "/Users/bojunfeng/cs/inference/xinference/core/utils.py", line 27, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bojunfeng/cs/inference/xinference/core/worker.py", line 182, in launch_builtin_model
subpool_address, devices = await self._create_subpool(model_uid, n_gpu=n_gpu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bojunfeng/cs/inference/xinference/core/worker.py", line 111, in _create_subpool
sub_pool_address = await self._main_pool.append_sub_pool(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MainActorPool' object has no attribute 'append_sub_pool'Since the model is not launched, later requests to create a Gradio page for the model also fails, causing the exact same error "model not found in model list" since the model is not launched at all and therefore does not exist, resulting in a "model not found" error page instead of the Gradio interface as expected.
Bojun-Feng
commented
11 months ago I have managed to solve this issue on my end, not sure if the root of the problem is the exact same, but I hope this information is helpful
TLDR: some dependency packages are not up to date, so reinstall Xinference with all dependencies
I noticed that the MainActorPool object is imported from another python module Xoscar, so if MainActorPool is causing the problem it is probably not Xinference's fault.
Took a look at setup.cfg:
install_requires =
xoscar>=0.1.2Running pip show xoscar:
Name: xoscar
Version: 0.1.0Steps I took to resolve the problem:
pip install -e ".[all]" (Full log with all updated modules provided below)xinference in terminal
YinSonglin1997
commented
11 months ago The problem seems to be relavent to
XINFERENCE_MODEL_SRC=xorbits. This env var force xinference downloading models from our s3 bucket, which is not accessible right now.We've added support of downloading from modelscope, and you can enable this feature by
XINFERENCE_MODEL_SRC=modelscope.
我在constants.py中设置参数XINFERENCE_MODEL_SRC=“modelscope”，但是Xinference的界面似乎并没有按照llm_family_modelscope.json改变，请问还需要设置什么参数吗？
Describe the bug
I have built a Docker image myself and deployed xinference on k8s. The homepage can be accessed normally. However, loading the model failed, and the error message is 'not found'. This error is consistently reproducible, making xinference basically unusable.
To Reproduce
To help us to reproduce this bug, please provide information below:
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
dockerfile
FROM python:3.10
ENV XINFERENCE_MODEL_SRC=xorbits
RUN pip install --upgrade pip setuptools -i https://pypi.tuna.tsinghua.edu.cn/simple/
RUN pip install "xinference[all]" -i https://pypi.tuna.tsinghua.edu.cn/simple/
EXPOSE 9997
CMD ["/usr/local/bin/xinference","-H 0.0.0.0", "-p 9997"]