HarmJ0y
commented
7 months ago - NLP container now splits large docs and submits them through an additional queue for embedding
- Added overall plaintext size limit that prevents larger docs from being split/indexed
- Memory and CPU limits added to NLP container to prevent starvation via the NLP container
- Average embedding vector approach eliminated due to the new streaming approach
- Additional model options pre-loaded for NLP container
- Changed default embedding model to TaylorAI/gte-tiny
- Vector normalization added as k8s options for the NLP container
- Removed /indexing web endpoint for NLP service as text should be submitted through the Nemesis frontend
- Tokenization fixed for NLP container - now using the chosen model's tokenizer
- Added plaintext file processing limits for elastic ingestion, embedding, and crack list building
- Added metadata filtering for all document searches