problem on execute the news searcah example

[zjbit]

I execute the news search example as following (1)pip install jina[all] (2)download the example data (3)extract the data to the /tmp dir (4)execute: python prepare.py (5)execute: python app.py -t index and got error as following: encoder driver throws an exception, the sequel pipeline may not work properly 🐳 encoder@ 8[I]:received "index" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸⚐ 🐳 encoder@ 8[I]:batching enabled for BaseTransformerEncoder.encode(). batch_size=32 num_batch=None axis=0 encoder@21899[I]:received "control" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐ encoder@21902[I]:received "index" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐ chunk_indexer@22043[E]:unknown exception: vectors shape 790 is not valid, expecting "vectors" to have rank of 2 Traceback (most recent call last): File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 352, in run self.loop_body() File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 317, in loop_body msg = self.zmqlet.recv_message(callback=self.msg_callback) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/zmq.py", line 222, in recv_message return callback(msg) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 301, in msg_callback return self._callback(msg) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 291, in _callback self.pre_hook(msg).handle(msg).post_hook(msg) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 179, in handle self.executor(self.request_type) File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/init.py", line 534, in call d() File "/usr/anaconda3/lib/python3.7/site-packages/jina/drivers/index.py", line 32, in call self.exec_fn(np.array([c.chunk_id for c in chunk_pts]), np.stack(embed_vecs)) File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/decorators.py", line 22, in arg_wrapper f = func(self, *args, kwargs) File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py", line 87, in add raise ValueError('vectors shape %s is not valid, expecting "vectors" to have rank of 2' % vectors.shape) ValueError: vectors shape 790 is not valid, expecting "vectors" to have rank of 2 chunk_indexer@22046[E]:unknown exception: vectors shape 1162 is not valid, expecting "vectors" to have rank of 2 Traceback (most recent call last): File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 352, in run self.loop_body() File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 317, in loop_body msg = self.zmqlet.recv_message(callback=self.msg_callback) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/zmq.py", line 222, in recv_message return callback(msg) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 301, in msg_callback return self._callback(msg) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 291, in _callback self.pre_hook(msg).handle(msg).post_hook(msg) File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 179, in handle self.executor(self.request_type) File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/init.py", line 534, in call d() File "/usr/anaconda3/lib/python3.7/site-packages/jina/drivers/index.py", line 32, in call self.exec_fn(np.array([c.chunk_id for c in chunk_pts]), np.stack(embed_vecs)) File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/decorators.py", line 22, in arg_wrapper f = func(self, *args, *kwargs) File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py", line 87, in add raise ValueError('vectors shape %s is not valid, expecting "vectors" to have rank of 2' % vectors.shape) ValueError: vectors shape 1162 is not valid, expecting "vectors" to have rank of 2 chunk_indexer@22043[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@22043[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@22043[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@22043[I]:dumped changes to the executor, 4s since last the save chunk_indexer@22043[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 287.5 KB chunk_indexer@22046[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@22046[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@22046[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@22046[I]:dumped changes to the executor, 4s since last the save chunk_indexer@22046[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 288.6 KB chunk_indexer@22046[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 288.6 KB chunk_indexer@22046[S]:terminated chunk_indexer@22043[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 287.5 KB chunk_indexer@22043[S]:terminated 🐳 encoder@ 8[E]: 🐳 Found no NVIDIA driver on your system. Please check that you 🐳 have an NVIDIA GPU and installed a driver from 🐳 http://www.nvidia.com/Download/index.aspx 🐳 Traceback (most recent call last): 🐳 File "/opt/conda/lib/python3.7/site-packages/jina/drivers/encode.py", line 30, in call 🐳 embeds = self.exec_fn(contents) 🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/decorators.py", line 177, in arg_wrapper 🐳 r = func(self, b, args, kwargs) 🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/decorators.py", line 53, in arg_wrapper 🐳 r = func(self, *args, **kwargs) 🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py", line 61, in encode 🐳 token_ids_batch = self.array2tensor(token_ids_batch) 🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py", line 250, in array2tensor 🐳 return tensor.cuda() if self.on_gpu else tensor 🐳 File "/opt/conda/lib/python3.7/site-packages/torch/cuda/init.py", line 149, in _lazy_init 🐳 _check_driver() 🐳 File "/opt/conda/lib/python3.7/site-packages/torch/cuda/init.py", line 54, in _check_driver 🐳 http://www.nvidia.com/Download/index.aspx""") 🐳 AssertionError: 🐳 Found no NVIDIA driver on your system. Please check that you 🐳 have an NVIDIA GPU and installed a driver from 🐳 http://www.nvidia.com/Download/index.aspx 🐳 encoder@ 8[W]:encoder driver throws an exception, the sequel pipeline may not work properly encoder@21899[I]:received "control" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐ encoder@21902[I]:received "index" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐

could you pleased to tell me what happen or how to overcome it, thank you.

[xiongma]

please format the log with markdown, I can't see it clearly

[zjbit]

i upload err msg as followed

thank

[zjbit]

i ran this demo on a centos CPU server(8C16G)

[xiongma]

this is because the model used gpu, wait a minute, I am rebuilding the image.

[zjbit]

OK

[xiongma]

finish rebuild, pull again

[zjbit]

ok，in addition, is the gpu and cpu are supproted by the new images at the same time？

[zjbit]

the error is exist as before the image i pull is not the newest(through : python app.py -t index)

[zjbit]

i tried just now, but it can't work, i posted the error msg, which shows the image pulled from docker is not the newest you rebuild yesterday, is there any other configuration should i do?

[xiongma]

sorry about that, we are detecting the problem, it will be soon

[zjbit]

OK

[nan-wang]

@policme, any progress on this?

[zjbit]

i tried other examples like webqa search, which can work properly, i compared its flow-index.yml with that of the news search example, and find the encoder part are different. (1)news-search encoder: image: jinaai/hub.executors.encoders.nlp.transformers-hitscir timeout_ready: 6000000 replicas: 2

(2)web-qa encoder: yaml_path: pods/encoder/encoder.yml timeout_ready: 60000 replicas: 3 read_only: True

is there any difference between the two encoders, i foud the encoder of the webqa example is with transformer_roberta, and the encoder of the news-search is a docker images which can't work properly on my system up to now.

[xiongma]

1. next time, if u want to put the code, please format the code with markdown, so we can see the meaning you want to give to me clearly.
2. because of some reason, the roberta docker image has some flaws, wait a while, we are fixing it

[zjbit]

ok，i mean the two jina examples, news-search and webqa, are with differnt encoders, the encoder of webqa was installed with jina through pip jina, and the encoder of new-search is from a docker of jina-huh, is that right?

[xiongma]

1. the encoder of webqa didin't install through jina, please refer this part
2. the encoder of news is same as webqa encoder, two of them use roberta as model, between them, the difference is that webqa's encoder is a py file or yaml file, the new's encoder is a docker image.
3. no matter webqa encoder or news encoder is not in jina built-in, both of them are customized。

[zjbit]

thank