Open Charlestar opened 1 year ago
Hi, what's your output of jina -vf
?
Can you set JINA_LOG_LEVEL=debug
and start the server again?
Hi, thank you for your reply! here is my jina -vf
output after set JINA_LOG_LEVEL=debug
. By restart the server, I find the problem is GRPC not ready (see the output below), then I try to install grpcio and grpcio-tools from conda, but it's still don't work, what should I do next?
(jina) istar@DESKTOP:~$ jina -vf
DeprecationWarning: pkg_resources is deprecated as an API (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:121)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2349)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
- jina 3.19.0
- docarray 0.21.1
- jcloud 0.2.12
- jina-hubble-sdk 0.39.0
- jina-proto 0.1.27
- protobuf 3.20.3
- proto-backend cpp
- grpcio 1.47.5
- pyyaml 6.0
- python 3.8.17
- platform Linux
- platform-release 5.15.90.1-microsoft-standard-WSL2
- platform-version #1 SMP Fri Jan 27 02:56:13 UTC 2023
- architecture x86_64
- processor x86_64
- uid 91769515676
- session-id 62682054-2529-11ee-ab13-00155de3ae9c
- uptime 2023-07-18T13:10:16.805832
- ci-vendor (unset)
- internal False
* JINA_DEFAULT_HOST (unset)
* JINA_DEFAULT_TIMEOUT_CTRL (unset)
* JINA_DEPLOYMENT_NAME (unset)
* JINA_DISABLE_UVLOOP (unset)
* JINA_EARLY_STOP (unset)
* JINA_FULL_CLI (unset)
* JINA_GATEWAY_IMAGE (unset)
* JINA_GRPC_RECV_BYTES (unset)
* JINA_GRPC_SEND_BYTES (unset)
* JINA_HUB_NO_IMAGE_REBUILD (unset)
* JINA_LOG_CONFIG (unset)
* JINA_LOG_LEVEL debug
* JINA_LOG_NO_COLOR (unset)
* JINA_MP_START_METHOD (unset)
* JINA_OPTOUT_TELEMETRY (unset)
* JINA_RANDOM_PORT_MAX (unset)
* JINA_RANDOM_PORT_MIN (unset)
* JINA_LOCKS_ROOT (unset)
* JINA_K8S_ACCESS_MODES (unset)
* JINA_K8S_STORAGE_CLASS_NAME (unset)
* JINA_K8S_STORAGE_CAPACITY (unset)
* JINA_STREAMER_ARGS (unset)
(jina) istar@DESKTOP:~$ python -m clip_server
DEBUG clip_t/rep-0@484 Setting signal handlers [07/18/23 13:23:05]
DEBUG clip_t/rep-0@484 Signal handlers already set
DEBUG clip_t-replica-set@480 Waiting for ReplicaSet to start successfully [07/18/23 13:23:05]
DEBUG gateway/rep-0@485 Setting signal handlers [07/18/23 13:23:05]
β Waiting ... ββββββββββββββββββββββββββββββββββββββββ 0/2 -:--:--DEBUG gateway/rep-0@485 Signal handlers already set
DEBUG gateway-replica-set@480 Waiting for ReplicaSet to start successfully [07/18/23 13:23:05]
β Waiting ... ββββββββββββββββββββββββββββββββββββββββ 0/2 -:--:--DEBUG gateway/rep-0@485 adding connection for deployment clip_t/heads/0 to [07/18/23 13:23:05]
grpc://0.0.0.0:60420
DEBUG gateway/rep-0@485 create_connection connection for clip_t to grpc://0.0.0.0:60420
DEBUG gateway/rep-0@485 create_connection connection for clip_t to grpc://0.0.0.0:60420
DEBUG gateway/rep-0@485 connection for deployment clip_t/heads/0 to grpc://0.0.0.0:60420
added
DEBUG gateway/rep-0@485 Setting up GRPC server
INFO gateway/rep-0@485 start server bound to 0.0.0.0:51000
DEBUG gateway/rep-0@485 Running GatewayRuntime warmup
DEBUG gateway/rep-0@485 starting warmup task for deployment clip_t
DEBUG gateway/rep-0@485 server bound to 0.0.0.0:51000 started
DEBUG gateway/rep-0@485 GRPC server setup successful
DEBUG gateway/rep-0@480 ready and listening [07/18/23 13:23:06]
DEBUG gateway-replica-set@480 ReplicaSet started successfully [07/18/23 13:23:06]
DEBUG gateway@480 Deployment started successfully [07/18/23 13:23:06]
β Ή Waiting clip_t gateway... ββββββββββββββββββββββββββββββββββββββββ 0/2 0:00:00UserWarning: `docs` annotation must be a type hint, got DocumentArray instead, you should maybe remove the string annotation. Default valueDocumentArray will be used instead. (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/jina/serve/executors/__init__.py:131)
β Waiting clip_t... βββββββββββββββββββββΊβββββββββββββββββββ 1/2 0:00:00DeprecationWarning: Importing from timm.models.layers is deprecated, please import via timm.layers (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/timm/models/layers/__init__.py:49)
β ΄ Waiting clip_t... βββββββββββββββββββββΊβββββββββββββββββββ 1/2 0:00:10DEBUG clip_t/rep-0@484 <clip_server.executors.clip_torch.CLIPEncoder object at [07/18/23 13:23:16]
0x7fd25709aee0> is successfully loaded!
DEBUG clip_t/rep-0@484 Setting up GRPC server
INFO clip_t/rep-0@484 start server bound to 0.0.0.0:60420
DEBUG clip_t/rep-0@484 server bound to 0.0.0.0:60420 started
DEBUG clip_t/rep-0@484 GRPC server setup successful
DEBUG clip_t/rep-0@480 Checking readiness to 0.0.0.0:60420 with protocol GRPC [07/18/23 13:23:16]
β Waiting clip_t... βββββββββββββββββββββΊβββββββββββββββββββ 1/2 0:00:10DEBUG clip_t/rep-0@484 recv _status request
DEBUG gateway/rep-0@485 completed warmup task in 10.483339071273804s. [07/18/23 13:23:16]
DEBUG clip_t/rep-0@480 Exception: <AioRpcError of RPC that terminated with: [07/18/23 13:23:17]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string =
"{"created":"@1689657797.204551546","description":"Deadline
Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81β¦
>
DEBUG clip_t/rep-0@480 Server on 0.0.0.0:60420 with protocol GRPC is not yet ready
DEBUG clip_t/rep-0@480 Checking readiness to 0.0.0.0:60420 with protocol GRPC
DEBUG clip_t/rep-0@480 Exception: <AioRpcError of RPC that terminated with: [07/18/23 13:23:18]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string =
"{"created":"@1689657798.313920705","description":"Deadline
Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81β¦
>
DEBUG clip_t/rep-0@480 Server on 0.0.0.0:60420 with protocol GRPC is not yet ready
... (repeat the GRPC not ready message)
Unfortunately, we currently do not have access to Windows devices for testing purposes. While we investigate this issue further, we recommend trying the following troubleshooting steps:
Clear the model cache: Clearing the model cache can help resolve any potential caching conflicts. It normally stores at ~/.cache
Configure the VPN: If you are using a VPN, ensure that it is properly configured and not causing any network-related issues. Double-check the VPN settings and try reconnecting or using a different VPN server if possible.
Use a Linux device or Docker image: If you have access to a Linux device or a Docker image, try running the application on that platform. This can help determine if the issue is specific to Windows or if it's a more general problem.
Utilize the SaaS CLIP service via inference.jina.ai: As an alternative, you can leverage our SaaS CLIP service for inference by visiting inference.jina.ai. This allows you to perform inference tasks without relying on local setups.
We appreciate your patience while we work on resolving this issue. If you have any further questions or concerns, please don't hesitate to reach out to us.
I have similar issue but seems related to some permission issue because I see some 403
when downloading the model
DEBUG gateway/rep-0@32033 starting warmup task for deployment clip_o
DEBUG gateway/rep-0@32033 server bound to 0.0.0.0:51000 started
DEBUG gateway/rep-0@32033 GRPC server setup successful
objc[32033]: +[__NSTimeZone initialize] may have been in progress in another thread when fork() was called.
objc[32033]: +[__NSTimeZone initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
DEBUG gateway/rep-0@32023 ready and listening [09/18/23 23:12:13]
DEBUG gateway-replica-set@32023 ReplicaSet started successfully
DEBUG gateway@32023 Deployment started successfully [09/18/23 23:12:13]
UserWarning: `docs` annotation must be a type hint, got DocumentArray instead, you should maybe remove the string annotation. Default valueDocumentArray will be used instead. (raised from /opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg/jina/serve/executors/__init__.py:265)
Failed to download
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual
.onnx with <HTTPError 403: 'Forbidden'> at the 0th attempt
Failed to download
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual
.onnx with <HTTPError 403: 'Forbidden'> at the 1th attempt
Failed to download
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual
.onnx with <HTTPError 403: 'Forbidden'> at the 2th attempt
textual.onnx 0.0% β’ 0/100 bytes β’ ? β’ -:--:--
CRITI⦠clip_o/rep-0@32032 can not load the executor from {"jtype": "CLIPEncoder", "metas": [09/18/23 23:12:15]
{"py_modules": ["clip_server.executors.clip_onnx"]}}
ERROR clip_o/rep-0@32032 RuntimeError('Failed to download
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276β¦
within retry limit 3') during 'WorkerRuntime' initialization
add "--quiet-error" to suppress the exception details
Traceback (most recent call last):
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 140, in run
runtime = AsyncNewLoopRuntime(
^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 90, in __init__
self._loop.run_until_complete(self.async_setup())
File
"/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11β¦
line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 270, in async_setup
self.server = self._get_server()
^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 192, in _get_server
return GRPCServer(
^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 34, in __init__
super().__init__(**kwargs)
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 56, in __init__
self._request_handler = req_handler or self._get_request_handler()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 81, in _get_request_handler
return self.req_handler_cls(
^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 139, in __init__
self._load_executor(
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 354, in _load_executor
self._executor: BaseExecutor = BaseExecutor.load_config(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 792, in load_config
obj = JAML.load(tag_yml, substitute=False, runtime_args=runtime_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 174, in load
r = yaml.load(stream, Loader=get_jina_loader_with_runtime(runtime_args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/yaml/__init__.py", line 81, in
load
return loader.get_single_data()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/yaml/constructor.py", line 51,
in get_single_data
return self.construct_document(node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/yaml/constructor.py", line 55,
in construct_document
data = self.construct_object(node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/yaml/constructor.py", line 100,
in construct_object
data = constructor(self, node)
^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 582, in _from_yaml
return get_parser(cls, version=data.get('version', None)).parse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.eggβ¦
line 46, in parse
obj = cls(
^^^^
File
Hi! I'm trying to deploy clip-as-service on my wsl2, which is Ubuntu 22.04.2 and with CUDA 12.2 for wsl installed. The python env has python=3.8 and pytorch=2.0.1. After I installed clip_server and clip_client, I run
python -m clip_server
as the README shows, however, it just download the 'ViT-B-32.pt' model and failed to continue. Here is the error log: Please tell me how to fix this problem, thanks!