search

jina-ai / jina

☁️ Build multimodal AI applications with cloud-native stack
https://docs.jina.ai
Apache License 2.0
20.98k stars 2.22k forks source link

Pod times out before model is downloaded. #395

Closed vforvoid closed 4 years ago

vforvoid commented 4 years ago

I'm running the example X as a service and using yaml to describe my encode pod. I want to download model from the web, so I put model_name: distilbert-base-cased in yaml. Encode pod has timeout_ready: 20000 parameter. It's not enough time for my machine to load the model, so my program fails with exception. 

         encode@8737[I]:post initiating, this may take some time...
Downloading:  54%|█████▎    | 141M/263M [00:17<00:14, 8.67MB/s]Traceback (most recent call last):
  File "/home/v/PycharmProjects/JinaEmbeddingServer/test2.py", line 26, in <module>
    with f:
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/flow/__init__.py", line 425, in __enter__
    return self.start()
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/flow/__init__.py", line 467, in start
    self._pod_stack.enter_context(v)
  File "/usr/lib/python3.8/contextlib.py", line 425, in enter_context
    result = _cm_type.__enter__(cm)
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/peapods/pod.py", line 233, in __enter__
    return self.start()
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/peapods/pod.py", line 370, in start
    return super().start()
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/peapods/pod.py", line 203, in start
    self.stack.enter_context(p)
  File "/usr/lib/python3.8/contextlib.py", line 425, in enter_context
    result = _cm_type.__enter__(cm)
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/peapods/pea.py", line 418, in __enter__
    return self.start()
  File "/home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/peapods/pea.py", line 414, in start
    raise TimeoutError(
TimeoutError: <class 'jina.peapods.pea.BasePea'> with name encode-1 can not be initialized after 20000.0ms

Process finished with exit code 1

Personally, I find it rather scary to use timeout_ready. How much time do I need to load and start the model? Is 20 second going to be enough? 60 seconds? Can I guarantee it? Sounds like I'm flipping a coin each time I'm starting my app. Is there any way to make it take its time to load everything and run whenever it's ready?

jina                          0.1.8
jina-proto                    0.0.20
jina-vcs-tag                  (unset)
libzmq                        4.3.2
pyzmq                         1.18.4
protobuf                      3.11.3
proto-backend                 cpp
grpcio                        1.28.1
ruamel.yaml                   0.16.10
python                        3.8.2
platform                      Linux
platform-release              5.4.0-29-generic
platform-version              #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020
architecture                  x86_64
processor                     x86_64
jina-resources                /home/v/PycharmProjects/JinaEmbeddingServer/venv/lib/python3.8/site-packages/jina/resources
JINA_ARRAY_QUANT              (unset)
JINA_CONTRIB_MODULE           (unset)
JINA_CONTRIB_MODULE_IS_LOADING(unset)
JINA_CONTROL_PORT             (unset)
JINA_DEFAULT_HOST             (unset)
JINA_EXECUTOR_WORKDIR         (unset)
JINA_FULL_CLI                 (unset)
JINA_IPC_SOCK_TMP             (unset)
JINA_LOG_FILE                 (unset)
JINA_LOG_LONG                 (unset)
JINA_LOG_NO_COLOR             (unset)
JINA_LOG_PROFILING            (unset)
JINA_LOG_SSE                  (unset)
JINA_LOG_VERBOSITY            (unset)
JINA_POD_NAME                 (unset)
JINA_PROFILING                (unset)
JINA_SOCKET_HWM               (unset)
JINA_STACK_CONFIG             (unset)
JINA_TEST_CONTAINER           (unset)
JINA_TEST_GPU                 (unset)
JINA_TEST_PRETRAINED          (unset)
JINA_VCS_VERSION              (unset)
JINA_VERSION                  (unset)
JINA_WARN_UNNAMED             (unset)
hanxiao commented 4 years ago

Hi, thanks for your feedback. You can set timeout_ready to -1, that will make a Pod to wait forever until everything is fully loaded.

The reason to have timeout_ready is that the Pod can be started either locally, inside a container, remotely, and remotely inside a container. A callback function is designed inside the Pod to acknowledge the spawner (e.g. the Flow) a successful start in all those scenarios. If the spawner can't get signal in time, then it considers something wrong on the Pod side and won't wait for it anymore.

vforvoid commented 4 years ago

Oh, ok, great, thanks and sorry for the trouble then!

It would be really nice to have this info in docs I've searched for timeout_ready and all three entries I found no such an explanation. Same goes for many other methods and its parameters in API docs. E.g. max_sent_len in Sentencizer.

max_sent_len (int) – the maximal length of the sentence.

It can state that Sentencizer will exclude all the sentences that are longer than max_sent_len symbols after stripping. Defaults to 1e5 if not set. It's not exactly obvious from the name if it's length in characters or tokens.

hanxiao commented 4 years ago

Aha, this is because timeout-ready is an argument defined at the CLI-level, it is phrased as timeout-ready not timeout_ready, so if you search for the former, it will show up. Related https://github.com/jina-ai/jina/issues/219

You probably notice that in Flow/Pod API you can use these CLI-level arguments in Python and YAML as well, by simply replacing the dash to underscore. This conversion is done implicitly via argparse.

Apparently, our sphinx is not smart enough to capture such implicit conversion. I will see what I can do to improve the doc indexing.

Can you look at this Sentencizer problem? @nan-wang

nan-wang commented 4 years ago

Oh, ok, great, thanks and sorry for the trouble then!

It would be really nice to have this info in docs I've searched for timeout_ready and all three entries I found no such an explanation. Same goes for many other methods and its parameters in API docs. E.g. max_sent_len in Sentencizer.

max_sent_len (int) – the maximal length of the sentence.

It can state that Sentencizer will exclude all the sentences that are longer than max_sent_len symbols after stripping. Defaults to 1e5 if not set. It's not exactly obvious from the name if it's length in characters or tokens.

hi, there. We've just updated the docs for Sentencizer in PR-409 Both min_sent_length and max_sent_length are the number of characters after stripping. Thanks a lot for pointing this out. The team is still in a tiny size and we definitely needs the OSS community to help us further improve the docs as well as the codes. Please feel free to make an issue or PR concerning any further problems during using jina.