Consolidate worker and vdb

dgarnitz / vectorflow

VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.

https://www.getvectorflow.com/

Apache License 2.0

676 stars 49 forks source link

Closed dgarnitz closed 8 months ago

dgarnitz commented 8 months ago

Consolidated the VDBUploaderWorker and Worker into one file. It was not necessary to have them be separate
Removed the HuggingFace embedding module. It was not being used
Added support for different OpenAI embedding models
Removed image search and embeddings code. They were not being used
Altered action workflow

Can see that the embedding and upload succeeds + tests