How and when do we process mumblings and build the vector database?

codemonger-io / mumble

Mumble in public through ActivityPub

MIT License

2 stars 1 forks source link

How and when do we process mumblings and build the vector database? #27

Open kikuomax opened 1 year ago

kikuomax commented 1 year ago

How about to process mumblings and rebuild the database every day until the update capability is implemented?

kikuomax commented 1 year ago

Recalculating embeddings for all the mumblings every day is waste of money. So we want to cache the embeddings.

kikuomax commented 1 year ago

How do we configure parameters for database building:

Number of partitions?
Number of codes (clusters)?

kikuomax commented 1 year ago

The initial version may be published with a database built offline with the experimental project mumble-embedding.

kikuomax commented 10 months ago

We want to index every mumbling when it is stored in the outbox folder. We want to progressively update the vector database with the planned new feature of flechasdb:

https://github.com/codemonger-io/flechasdb/issues/11