hanxiao
commented
4 years ago I notice that the following warning appears quite often in your log.
splitter@8961[W]:doc 42 gives no chunkAs it suggests, this doc contains no chunk. What does that mean to the whole pipeline? Let's say your batch_size=2, then each of your request has two documents. If you happens to have 2 empty-chunk-doc in a row. Then the encoder will have nothing to encode with: it receives empty chunks and returns None. The error ValueError('need at least one array to stack') is no surprising then, as there is no embedding generated for this batch.
Quick fix suggested for you and for now:
- Change your crafter parameter and make the chunk-generation less restrictive, say increase the
max_sent_length - Increase the
batch_sizeand make it more unlikely to encounter this problem.
And yes, a real fix is required on our side to ensure the workflow is robust regardless empty batch or not.
vitojph
nan-wang
Describe your problem
I'm trying to reproduce the BERT-based Semantic Search Engine with a different collection. Unlike the SouthPark example, my corpus is made of short text documents, a couple of paragraphs long. I preprocessed my collection to segment the sentences using spaCy, and generate a single two-column CSV file with the following structure:
doc_id, textWhat is your guess?
Whenever I try to index this collection, the process gets stuck after a
ValueError: need at least one array to stackerror:Click here to see the complete logs
``` $ python app.py -t index -n 100 Flow@8953[S]:successfully built Flow from a yaml config Sentencizer@8961[I]:post initiating, this may take some time... Sentencizer@8961[I]:post initiating, this may take some time takes 0.001 secs Sentencizer@8961[S]:successfully built Sentencizer from a yaml config splitter@8961[I]:setting up sockets... splitter@8961[I]:input tcp://0.0.0.0:56579 (SUB_CONNECT) output tcp://0.0.0.0:39789 (PUSH_CONNECT) control over tcp://0.0.0.0:42053 (PAIR_BIND) splitter@8961[S]:ready and listening TransformerTorc@8964[I]:post initiating, this may take some time... TransformerTorc@8964[I]:post initiating, this may take some time takes 2.714 secs TransformerTorc@8964[S]:successfully built TransformerTorchEncoder from a yaml config encoder@8964[I]:setting up sockets... encoder@8964[I]:input tcp://0.0.0.0:39789 (PULL_BIND) output tcp://0.0.0.0:51541 (PUSH_CONNECT) co ntrol over tcp://0.0.0.0:35377 (PAIR_BIND) encoder@8964[S]:ready and listening NumpyIndexer@8969[I]:post initiating, this may take some time... NumpyIndexer@8969[I]:post initiating, this may take some time takes 0.001 secs NumpyIndexer@8969[S]:successfully built NumpyIndexer from a yaml config BasePbIndexer@8969[I]:post initiating, this may take some time... BasePbIndexer@8969[I]:post initiating, this may take some time takes 0.000 secs BasePbIndexer@8969[S]:successfully built BasePbIndexer from a yaml config ChunkIndexer@8969[I]:post initiating, this may take some time... ChunkIndexer@8969[I]:post initiating, this may take some time takes 0.000 secs ChunkIndexer@8969[S]:successfully built ChunkIndexer from a yaml config NumpyIndexer@8969[W]:you can not query fromWhen I press ctrl-c, the logs continue:
Click here
``` ^C [1486.595 secs] ✅ done in ⏱ 1486.6s 🐎 0.0/s chunk_indexer@8969[W]:user cancel the process doc_indexer@8972[W]:user cancel the process join_all@8975[W]:user cancel the process DocPbIndexer@8972[I]:no update since 2020-05-06 13:41:36, will not save. If you really want to save it, call "touch()" before "save()" to force savin g doc_indexer@8972[I]:executor says there is nothing to save PyClient@8953[W]:user cancel the process doc_indexer@8972[I]:#sent: 0 #recv: 0 sent_size: 0 Bytes recv_size: 0 By tes join_all@8975[I]:#sent: 0 #recv: 3 sent_size: 0 Bytes recv_size: 1.7 KB splitter@8961[W]:user cancel the process splitter@8961[I]:#sent: 50 #recv: 50 sent_size: 26.9 KB recv_size: 20.9 KB NumpyIndexer@8969[S]:artifacts of this executor (vecidx) is persisted to /home/aiteam/projects/jina-examples/test/sbnpsago/chunk_indexer-0/vecidx.bin BasePbIndexer@8969[S]:artifacts of this executor (chunkidx) is persisted to /home/aiteam/projects/jina-examples/test/sbnpsago/chunk_indexer-0/chunkidx.bin PyClient@8953[S]:terminated join_all@8975[I]:#sent: 0 #recv: 3 sent_size: 0 Bytes recv_size: 1.7 KB ChunkIndexer@8969[I]:no update since 2020-05-06 13:41:36, will not save. If you really want to save it, call "touch()" before "save()" to force saving chunk_indexer@8969[I]:dumped changes to the executor, 1487s since last the save join_all@8975[S]:terminated chunk_indexer@8969[I]:#sent: 3 #recv: 3 sent_size: 1.7 KB recv_size: 10.9 KB splitter@8961[I]:#sent: 50 #recv: 50 sent_size: 26.9 KB recv_size: 20.9 KB splitter@8961[S]:terminated GatewayPea@8953[S]:terminated doc_indexer@8972[I]:#sent: 0 #recv: 0 sent_size: 0 Bytes recv_size: 0 Bytes doc_indexer@8972[S]:terminated ```Environment
The example with the SouthPark documents works. Any idea of what's going on? Thanks in advance.