Add tags for embedded data

[DanilTk]

I've came over the following issue: E.g. when feeding my vector db with multiple pdfs on different topics, e.g. K8S and some medical related topic brochure and asking LLM smth - the one started hallucinating mixing topics from K8S and medical brochure.

In this regard a request would be to add tags so that it is possible to give hints to LLM on which context the question relates to

[ThomasVitale]

The DocumentReaderfunctionality has general support for adding metadata to a document during the loading phase. However, not all implementation allow adding custom metadata.

• TextReader -> Supported (see example)
• JsonReader -> Supported (see example)
• PagePdfDocumentReader and PagePdfDocumentReader -> No custom metadata (only default as shown here and here)
• TikaDocumentReader -> No custom metadata (only default as shown here)

When retrieving relevant documents, for supported vector stores (like Chroma or Weaviate), you can use filter expressions to include only documents labelled with certain metadata (example).