search

jina-ai / serve

☁️ Build multimodal AI applications with cloud-native stack
https://jina.ai/serve
Apache License 2.0
21.12k stars 2.22k forks source link

Run Jina examples with some errors #1295

Closed leonwanghui closed 3 years ago

leonwanghui commented 3 years ago

Describe your problem

I was trying to execute the jina examples of mindspore-lenet with jinahub, after getting some help from John, I modified the code and push it to my personal code repo. But when I tried to execute python app.py -t index, some errors occurred with the log below:

           Flow@8096[S]:successfully built Flow from a yaml config
           load@8104[I]:post initiating, this may take some time...
           load@8104[I]:post initiating, this may take some time takes 0 seconds (0.00s)
           load@8104[S]:successfully built BaseExecutor from a yaml config
           load@8104[I]:setting up sockets...
           load@8104[I]:input tcp://0.0.0.0:44723 (PULL_BIND)    output tcp://0.0.0.0:57413 (ROUTER_BIND)     control over tcp://0.0.0.0:52693 (PAIR_BIND)
           load@8104[S]:ready and listening
           load@8109[I]:post initiating, this may take some time...
           load@8109[I]:post initiating, this may take some time takes 0 seconds (0.00s)
           load@8109[S]:successfully built BaseExecutor from a yaml config
           load@8109[I]:setting up sockets...
           load@8109[I]:input tcp://0.0.0.0:54427 (PULL_BIND)    output tcp://0.0.0.0:58835 (PUSH_CONNECT)    control over tcp://0.0.0.0:39201 (PAIR_BIND)
           load@8109[S]:ready and listening
           load@8114[C]:unknown exception: ConstructorError(None, None, "could not determine a constructor for the tag '!MnistImageReader'", <_ruamel_yaml.Mark object at 0x7f52f54bb6b0>)
Traceback (most recent call last):
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 348, in run
    self.loop_body()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 329, in loop_body
    self.load_executor()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 200, in load_executor
    self.args.separated_workspace, self.args.pea_id)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/__init__.py", line 433, in load_config
    return yaml.load(tmp_s)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
    data = self.construct_object(node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 743, in construct_undefined
    node.start_mark,
ruamel.yaml.constructor.ConstructorError: could not determine a constructor for the tag '!MnistImageReader'
  in "<unicode string>", line 1, column 1
           load@8114[S]:terminated
           load@8096[C]:fail to start <class 'jina.peapods.pea.BasePea'> with name load-1, this often means the executor used in the pod is not valid
Traceback (most recent call last):
  File "app.py", line 59, in <module>
    main()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "app.py", line 43, in main
    with f:
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/flow/__init__.py", line 444, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/flow/__init__.py", line 500, in start
    self.enter_context(v)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 254, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 388, in start
    return super().start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 224, in start
    self.enter_context(p)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 425, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 418, in start
    raise PeaFailToStart
jina.excepts.PeaFailToStart

Would be appreciated if anyone could offer some help for trouble-shooting, thanks!

What is your guess?

Firstly I think it's the problem of version inconsistency, but even if I upgraded the jina[hub] version to 0.7.0, the error is still there.

Maybe it's because the code has been deprecated and needs some update, but considering I'm new to this project, it would be difficult to debug and update the source code.

Environment

jina 0.7.0 jina-proto 0.0.65 jina-vcs-tag (unset) libzmq 4.3.2 pyzmq 1.18.4 protobuf 3.13.0 proto-backend cpp grpcio 1.32.0 ruamel.yaml 0.16.12 python 3.7.5 platform Linux platform-release 4.15.0-65-generic platform-version #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 architecture x86_64 processor x86_64 jina-resources /root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/resources JINA_ARRAY_QUANT (unset) JINA_BINARY_DELIMITER (unset) JINA_CONTRIB_MODULE (unset) JINA_CONTRIB_MODULE_IS_LOADING(unset) JINA_CONTROL_PORT (unset) JINA_DB_COLLECTION (unset) JINA_DB_HOSTNAME (unset) JINA_DB_NAME (unset) JINA_DB_PASSWORD (unset) JINA_DB_USERNAME (unset) JINA_DEFAULT_HOST (unset) JINA_DISABLE_UVLOOP (unset) JINA_EXECUTOR_WORKDIR (unset) JINA_FULL_CLI (unset) JINA_IPC_SOCK_TMP (unset) JINA_LOG_CONFIG (unset) JINA_LOG_NO_COLOR (unset) JINA_POD_NAME (unset) JINA_PROFILING (unset) JINA_RANDOM_PORTS (unset) JINA_SOCKET_HWM (unset) JINA_TEST_GPU (unset) JINA_TEST_PRETRAINED (unset) JINA_VCS_VERSION (unset) JINA_WARN_UNNAMED (unset)

Screenshots

JoanFM commented 3 years ago

Hey @leonwanghui, first problem is you need to update the import of ImageReader in your lenet_mindspore.py.

Change

from jina.executors.crafters.image.io import ImageReader

to

from jina.hub.crafters.image.ImageReader import ImageReader
JoanFM commented 3 years ago

also you need to move the import of local lenet: From:

from ..lenet import LeNet5
import sys
sys.path.append('..')

To

import sys
sys.path.append('..')
from lenet import LeNet5
JoanFM commented 3 years ago

then make sure that you have all the installed prerequisites for these lenet.py file

leonwanghui commented 3 years ago

@JoanFM Thanks for the help, and it works to remove the error msg, but another error occurred with the log below:

         encode@8697[I]:setting up sockets...
         encode@8697[I]:input tcp://0.0.0.0:33305 (PULL_BIND)    output tcp://0.0.0.0:39185 (ROUTER_BIND)     control over tcp://0.0.0.0:36137 (PAIR_BIND)
         encode@8697[S]:ready and listening
         encode@8702[I]:post initiating, this may take some time...
         encode@8702[I]:post initiating, this may take some time takes 0 seconds (0.00s)
         encode@8702[S]:successfully built BaseExecutor from a yaml config
         encode@8702[I]:setting up sockets...
           JINA@8702[E]:error when binding port 54279 to 0.0.0.0
         encode@8702[I]:#sent: 0 #recv: 0 sent_size: 0 Bytes recv_size: 0 Bytes
         encode@8702[E]:failed to stop. 'ZmqStreamlet' object has no attribute 'io_loop'
         encode@8702[C]:zmqlet can not be initiated
         encode@8702[S]:terminated
         encode@8617[C]:fail to start <class 'jina.peapods.tail_pea.TailPea'> with name encode-tail, this often means the executor used in the pod is not valid
Traceback (most recent call last):
  File "app.py", line 59, in <module>
    main()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "app.py", line 43, in main
    with f:
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/flow/__init__.py", line 444, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/flow/__init__.py", line 500, in start
    self.enter_context(v)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 254, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 388, in start
    return super().start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 210, in start
    self.enter_context(p)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 425, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 418, in start
    raise PeaFailToStart
jina.excepts.PeaFailToStart
cristianmtr commented 3 years ago

encode@8617[C]:fail to start <class 'jina.peapods.tail_pea.TailPea'> with name encode-tail, this often means the executor used in the pod is not valid

Looks like you maybe provided a custom name somewhere in your .yml files where you shouldn't have.

Can you post your configs?

JoanFM commented 3 years ago

can you try putting a higher timeout_ready setting to a larger value for your encode pod?

leonwanghui commented 3 years ago

encode@8617[C]:fail to start <class 'jina.peapods.tail_pea.TailPea'> with name encode-tail, this often means the executor used in the pod is not valid

Looks like you maybe provided a custom name somewhere in your .yml files where you shouldn't have.

Can you post your configs?

@cristianmtr The config could be found at: https://github.com/leonwanghui/mindspore-jina-apps/tree/main/lenet-fashion_mnist/pods

leonwanghui commented 3 years ago

can you try putting a higher timeout_ready setting to a larger value for your encode pod?

@JoanFM I'm afraid the query operation hasn't been executed, since the error has occurred when I run python app.py -t index.

JoanFM commented 3 years ago

yes, but the encoding is independent, it is also used during index operations

leonwanghui commented 3 years ago

yes, but the encoding is independent, it is also used during index operations

@JoanFM After setting timeout_ready to 100000, the zmqlite error was gone, but instead there comes another error in index.yml:

          index@11437[I]:post initiating, this may take some time...
          index@11437[I]:post initiating, this may take some time takes 0 seconds (0.00s)
          index@11437[S]:successfully built BaseExecutor from a yaml config
          index@11437[I]:setting up sockets...
          index@11437[I]:input tcp://0.0.0.0:44493 (PULL_BIND)   output tcp://0.0.0.0:48641 (ROUTER_BIND)     control over tcp://0.0.0.0:50361 (PAIR_BIND)
          index@11437[S]:ready and listening
          index@11442[I]:post initiating, this may take some time...
          index@11442[I]:post initiating, this may take some time takes 0 seconds (0.00s)
          index@11442[S]:successfully built BaseExecutor from a yaml config
          index@11442[I]:setting up sockets...
          index@11442[I]:input tcp://0.0.0.0:43925 (PULL_BIND)   output tcp://0.0.0.0:39125 (PUSH_BIND)  control over tcp://0.0.0.0:38411 (PAIR_BIND)
          index@11442[S]:ready and listening
          index@11447[I]:post initiating, this may take some time...
          index@11447[I]:post initiating, this may take some time takes 0 seconds (0.00s)
          index@11447[S]:successfully built NumpyIndexer from a yaml config
          index@11447[I]:post initiating, this may take some time...
          index@11447[I]:post initiating, this may take some time takes 0 seconds (0.00s)
          index@11447[S]:successfully built BinaryPbIndexer from a yaml config
          index@11447[I]:post initiating, this may take some time...
          index@11447[I]:post initiating, this may take some time takes 0 seconds (0.00s)
          index@11447[S]:successfully built CompoundIndexer from a yaml config
          index@11447[S]:memmap is enabled for /tmp/jina/fashion_mnist/gaqxbfha/chunk_indexer-1/vec.gz
          index@11447[C]:unknown exception: TypeError("unsupported operand type(s) for *: 'int' and 'NoneType'")
Traceback (most recent call last):
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 348, in run
    self.loop_body()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 329, in loop_body
    self.load_executor()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 200, in load_executor
    self.args.separated_workspace, self.args.pea_id)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/__init__.py", line 433, in load_config
    return yaml.load(tmp_s)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
    data = self.construct_object(node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/compound.py", line 352, in from_yaml
    obj.components = lambda: data['components']
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/compound.py", line 258, in components
    self._set_routes()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/compound.py", line 308, in _set_routes
    for m, _ in inspect.getmembers(c, predicate=inspect.ismethod):
  File "/root/miniconda3/envs/mindspore/lib/python3.7/inspect.py", line 341, in getmembers
    value = getattr(object, key)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/helper.py", line 667, in __get__
    value = obj.__dict__[f'CACHED_{self.func.__name__}'] = self.func(obj)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/indexers/__init__.py", line 83, in query_handler
    r = self.get_query_handler()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/indexers/vector.py", line 126, in get_query_handler
    vecs = self.raw_ndarray
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/helper.py", line 667, in __get__
    value = obj.__dict__[f'CACHED_{self.func.__name__}'] = self.func(obj)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/executors/indexers/vector.py", line 157, in raw_ndarray
    return np.memmap(self.index_abspath, dtype=self.dtype, mode='r', shape=(self.size, self.num_dim))
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/numpy/core/memmap.py", line 245, in __new__
    size *= k
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
          index@11447[S]:terminated
          index@11315[C]:fail to start <class 'jina.peapods.pea.BasePea'> with name index-1, this often means the executor used in the pod is not valid
Traceback (most recent call last):
  File "app.py", line 58, in <module>
    main()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "app.py", line 42, in main
    with f:
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/flow/__init__.py", line 444, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/flow/__init__.py", line 500, in start
    self.enter_context(v)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 254, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 388, in start
    return super().start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pod.py", line 224, in start
    self.enter_context(p)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 425, in __enter__
    return self.start()
  File "/root/miniconda3/envs/mindspore/lib/python3.7/site-packages/jina/peapods/pea.py", line 418, in start
    raise PeaFailToStart
jina.excepts.PeaFailToStart

Thanks!