bugfx/int-envvvars - Githubissues

i-dot-ai / redbox

Bringing Generative AI to the way the Civil Service works

https://i-dot-ai.github.io/redbox/

MIT License

93 stars 31 forks source link

bugfx/int-envvvars #1004

Closed gecBurton closed 2 months ago

gecBurton commented 2 months ago

Context

As a PM I want the file ingest to try and load my file once, fail immediately if it cant, and let other files through.

The cause of our ingest failing is Azure rate limiting not Unstructured

Previously we were:

retying the Azure embedding 10 times from within langhanin
retrying the ingest indefinitely from Django-worker

Changes proposed in this pull request

The rerties have been set to 1
The time to wait between retries has been set to double the Azure suggested wait time
The chunk size has ben increased
The log levels have been set to INFO so that the logs are usable

Guidance to review

try and process large number of documents and note that:

generally they all get processed
in the event of failure the cause of failure is recorded in the admin

Relevant links

Things to check

[ ] I have added any new ENV vars in all deployed environments
[ ] I have tested any code added or changed
[ ] I have run integration tests