Currently, the index is stored as a vector JSON-file locally. So when the index is created with /job, the JSON has then to be manually copied to /backend and will then be pushed and also deployed to the Fly.io backend hosting.
This process is very bad because it's not very flexible and if we want to update the index, a new backend version has to be deployed. Also, having the index in the source is not optimal as well.
Currently, the index is stored as a vector JSON-file locally. So when the index is created with
/job, the JSON has then to be manually copied to/backendand will then be pushed and also deployed to the Fly.io backend hosting.This process is very bad because it's not very flexible and if we want to update the index, a new backend version has to be deployed. Also, having the index in the source is not optimal as well.
Two possible solution are Pinecone or Weaviate.
Going forward with Qdrant
https://cloud.qdrant.io/
/jobto store generated index on Qdrant. https://gpt-index.readthedocs.io/en/latest/how_to/vector_stores.html