search

valeriansaliou / sonic

🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
https://crates.io/crates/sonic-server
Mozilla Public License 2.0
20.11k stars 578 forks source link

How to know that the indexing has been completed? #285

Closed lixianwa closed 2 years ago

lixianwa commented 2 years ago

Thanks for your great work！ I have push hundreds of millions data to sonic，but can not search some terms，i dont know why， i guess the reason is that the indexing not completely. How to know that the indexing has been completed?

valeriansaliou commented 2 years ago

Hey! Thank you. Did you run a TRIGGER CONSOLIDATE pass? This is also done for you if you stop Sonic gracefully (the stop operation should then take some time if what you ingested was not consolidated), and start it again.

See: https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL.md#5%EF%B8%8F⃣-sonic-channel-control-mode

lixianwa commented 2 years ago

@valeriansaliou Thank you for reply！ I do TRIGGER CONSOLIDATE, but it not work for me. my step is: connect to Sonic, then BulkPush（go client） 10000 records every time ，not return error，total push about 60 million data （only call ）， then i find that can not search some terms, later I run TRIGGER CONSOLIDATE, But it not work still。whether my step is wrong for import mass data？

valeriansaliou commented 2 years ago

Could those terms be stopwords? How many terms per object-id did you push? (note that there is a configurable limit for terms per object in Sonic configuration)

lixianwa commented 2 years ago

Im sure it's not stopwords. maybe default configuire retain_word_objects is too small，I am trying set it large。 @valeriansaliou

lixianwa commented 2 years ago

Now i am sure the issus is due to retain_word_objects too small. Thanks for help