Add file name to chunk metadata and insert into formatted chunk - Githubissues

i-dot-ai / redbox

Bringing Generative AI to the way the Civil Service works

https://i-dot-ai.github.io/redbox/

MIT License

83 stars 27 forks source link

Add file name to chunk metadata and insert into formatted chunk #794

Closed wpfl-dbt closed 1 month ago

wpfl-dbt commented 1 month ago

Context

REDBOX-433. The filename is important metadata for the LLM. This stores it in Elastic then inserts it into the formatted retrieved document.

Changes proposed in this pull request

Change document loader to add a new metadata field: file_name
Change document formatter to pick up file_name if it exists

Guidance to review

Confirm you're happy with my changes to the document XML formatting

Things to check

[x] I have added any new ENV vars in all deployed environments
[x] I have tested any code added or changed
[x] I have run integration tests

lmwilkigov commented 1 month ago

LGTM