dluc
commented
2 weeks ago hi @roldengarm could you provide some more information on the following points?
- Name of the model used for embeddings.
- Embedding vector size (256, 512, 1024, 1536, or 3072).
- Are you truncating embedding vectors?
It seems the VM hosting Postgres might be running out of memory due to the vector index size. Here are a few options to consider:
- Disable vector indexing: Import all the data first and enable indexing afterward. Monitor the memory usage during this process and see if the indexing operation fails. Can you increase the memory if needed?
- Change index type: Consider using HNSW instead of IVFFlat. HNSW uses less memory, allowing you to index more records, though it may slow down search performance.
- Truncate embedding vectors: If you’re using
text-embedding-3, truncating the vectors can reduce the memory footprint. Keep in mind that this reduces also the precision of relevance scores: test this approach to see if Search returns incorrect results or if Ask produces hallucinations. For more details see "shortening embeddings" (https://openai.com/index/new-embedding-models-and-api-updates/) and MRL (e.g. here https://aniketrege.github.io/blog/2024/mrl/).
roldengarm
Context / Scenario
Running KM as a service, using queues, Azure OpenAI Embedding-3, PostgresDB as backend. Ingested about 900k records & ~15Gb of data. Consuming it from a console application using the WebClient.
Initially after a couple of thousand documents, when calling SearchAsync / AskAsync, it worked fine. However, after ~900k records, I'm getting a server error (500) every time. In the logs I can see a Npgsql.NpgsqlException - Timeout.
What happened?
I'm getting a server error every time. The ingestion process is still running at about ~12 documents at the same time. I don't see high CPU usage on the App Service or Postgres (Flexible Azure server). Tried increasing Postgres to 4vCores 16Gb RAM, no difference.
We're planning to ingest a total of 9m documents so it's concerning it's already throwing errors at 900k.
Importance
I cannot use Kernel Memory
Platform, Language, Versions
Using C#, KM deployed as a service to Azure App Service, using Azure Postgres Flexible Server, Azure Storage, and Azure OpenAI
This is our second try; we initially tried using Azure AI Search instead of Azure Postgres, but the costs for storing ~9m records was astronomical.
Relevant log output