VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.
670
stars
47
forks
source link
Open Source Embeddings with Queue #44
Closed
dgarnitz closed 1 year ago
What
refactored the worker and hugging face model embeddings to use a worker and rabbit mq rather than function like an API
Why
More scalable
Verification
Can see that the embedding requests succeed locally in Docker:
Can also see the inference occurring in the logs of the model container:
And now Rabbit MQ creates a queue for the model-name:
Unit Tests
Existing tests pass in the worker:
Although there are no tests for the HF inference app and the HF pathway inside the worker is not covered by existing tests.