search

datacommonsorg / website

Code for the Data Commons website
https://datacommons.org
Apache License 2.0
20 stars 74 forks source link

[NL param] Use a list of index types everywhere #4351

Closed shifucun closed 3 weeks ago

shifucun commented 3 weeks ago

NL server now allows a list of indexes. With this change, we don't need to merge index during embedding build time, but rather include them in the runtime.

This allows not replicate the same embeddings in multiple indexes and makes embedding build atomic for each source folder.

shifucun commented 3 weeks ago

Very nice!

Should we have an integration test with 2 idx= entries and do a correlation?

I will add one after cleaning up all current indexes, since they mostly include some kind of base index