search

opendistro-for-elasticsearch / k-NN

🆕 A machine learning plugin which supports an approximate k-NN search algorithm for Open Distro.
https://opendistro.github.io/
Apache License 2.0
277 stars 55 forks source link

How much memory occupied if index of hnsw file all loaded into memory? #300

Closed wrkaiser closed 3 years ago

wrkaiser commented 3 years ago

Hi， My index has merged one segment in one shard, and ".hnsw" file size is 2.1g，i want to know whether has statistics that are completely loaded into memory?

thanks.

wrkaiser commented 3 years ago

when i use warm up api , the error message is as follows: image

jmazanec15 commented 3 years ago

Hi @wrkaiser

/_opendistro/_stats will tell you what has been loaded into memory.

The failure for the warmup API is strange. Could you provide the following information?

  1. ODFE version number
  2. ODFE artifact type (Docker, RPM, TAR, etc)

Additionally, could you run the command: /_cat/plugins and paste the output?

wrkaiser commented 3 years ago

thanks，i get.

I should be using the oldest version that is 1.4.0.0.

wrkaiser commented 3 years ago

hi， Does the release version not provide packages that can be used directly? QQ截图20210113165932

wrkaiser commented 3 years ago

hi @jmazanec15, Coulde provide can directly used package in the lastest version?

thanks.

wrkaiser commented 3 years ago

hi，@vamshin, My network has wall，download dependent package is so slowly. I'm very much hope that you can provide a package that can be used directly.

thanks.

jmazanec15 commented 3 years ago

Hi @wrkaiser ,

Apologies, are you using ODFE 1.4?

wrkaiser commented 3 years ago

hi @jmazanec15,

Yes, i plan to upgrade to the latest version.

jmazanec15 commented 3 years ago

Oh okay thanks @wrkaiser. So which packages do you need? opendistro-knn-1.12.0.0.zip and libKNNIndexV206.so?

wrkaiser commented 3 years ago

@jmazanec15， yes~!

jmazanec15 commented 3 years ago

You should be able to get them from the tar ball: https://opendistro.github.io/for-elasticsearch-docs/docs/install/tar/

curl https://d3g5vo6xdbdb9a.cloudfront.net/tarball/opendistro-elasticsearch/opendistroforelasticsearch-1.12.0.tar.gz -o opendistroforelasticsearch-1.12.0.tar.gz

cd plugins/opendistro-knn/

ls
ls knn-lib

Please let me know if that does not work.