search

marella / chatdocs

Chat with your documents offline using AI.
MIT License
684 stars 99 forks source link

Memory optimization #14

Open MichaelMurithi opened 1 year ago

MichaelMurithi commented 1 year ago

Out of curiosity, I tried containerizing the package with the offline models, installing the required PIP packages and a sample document (size: 62 kB) loaded through chatdocs add.

The build could not complete locally due to memory issues. I tried it in a hosted environment and I got the following warnings during the build process:

docker_build_error server_health_error

Even though the package does not have an official support for containers yet, this indicates that we need to optimize the code perfoming various operations (like adding files or loading models) so that it works well on low end PCs.

marella commented 1 year ago

The default models in chatdocs.yml require at least 8 GB RAM to run. I picked the smallest models possible that have decent quality. You can use models smaller than the default ones by adding them to your chatdocs.yml file but the response quality might not be good. So it is a trade-off between quality and performance.