JoanFM
commented
4 years ago Hello Jose Manuel,
Thank you for opening the issue and happy to see your interest in jina.
Taking a look to your configuration I have some comments although I am not sure they are the root cause of your problem (Some of them are just recommendations). It would be good that you can share some more detailed logs so that we can take a closer look at them.
-
In your CustomModel, you are doing a
hub.load()call inside theencodemethod, which should be done at _post_init() step. Otherwise u are loading the model for every batch (I guess it is somehow bypassed and may not have huge impact but I would recommend not to call the load so often). -
A batch size of 1000 seems like a really big batch size that could affect the memory required by the
query encoderto work. -
Another suggestion is that you can avoid having 2 indexers (index-chunk) and (index-doc) and keep only one (the CompoundIndexer). In your case, you are not breaking the documents (text) in different chunks. Therefore u can keep a single
indexer(the index-chunk used now (Although you can name itindex-docsince in your case nochunkingis involved)). -
(For future work)When your dataset will grow, I would recommend trying different indexers different than
NumpyIndexersince it performs an exhaustive search that does not scale well for large amounts of data (see FaissIndexer, AnnoyIndexer ...)
These are just comments that may help you with your problem but I would also ask you to provide some more details of the logs.
I hope you find it helpful!
jmferrete
hanxiao
sync-by-unito[bot]
Describe your problem We are building a semantic text search system, using NumpyIndexer and cosine distance metric. We have been testing Jina with smaller sets for the moment, running in single instances, but our final goal is to run Jira in a distributed system over datasets with billions of documents.
As a first step, and to validate our configuration when querying smaller datasets, we have indexed around 30 million text documents (simple sentences), using the following settings:
(29889733 docs :white_check_mark: done in :stopwatch: 6595.7s :racehorse: 4531.8/s) So the indexing part is fine.
But we are having problems while performing queries over this data. Any query causes an out of memory error at OS level.
We tried running queries on differents environments:
But we are still having the same memory problems.
We would like to get some feedback about the problems we are having.
Dataset details: Regarding the input data (around 150GB of plain text files), it is spread into multiple files in the
./source/folder. Each input file has the following structure: file1file2
Basically, multiline json files, where each line is a json document. And each document is formed by two fields:
text: The text string (to be added as meta-information during indexing).vector: A pre-generated [512,]float32array generated from thetextfield by using the universal sentence encoder ("https://tfhub.dev/google/universal-sentence-encoder/4")Once the data is indexed, the resulting compressed dataset is around 56GB.
Here you can find the setup files tar.gz
What is your guess? We guess there must be something wrong in our configuration, as we didn’t expect to run into memory problems, given the dataset size and the instances we have used for testing.
Environment We have tried with differents environments, but the common one is this:
Screenshots
Extracted from logs:
┆Issue is synchronized with this Jira Task by Unito