Closed vladosam closed 1 month ago
Hey, thank you for reporting this. I will check what is happening with the GitHub loader
I have released a new version of the GitHub loader which can handle larger repositories. However, I do not recommend using al-minlm-l6-v2 for it. Instead, try using Nomic or MxBai via Ollama for local embedding. Please let me know if you still face any errors.
I test it already. :D But I test it with al-minlm-l6-v2 and it happened again, all available memory is used and all free space on hdd. I will try with ollama embedding. Thanks.
Success, now it works with Ollama Nomic embedding. It was super slow, over 3 hours to process github repo. But now it works with github sources. Thanks.
Thanks for reporting back. I will add a concurrency feature to avoid this slow embedding for larger data sources.
When I try to add bigger github repo like
![Screenshot from 2024-07-09 08-15-34](https://github.com/n4ze3m/dialoqbase/assets/8487479/38d221ba-5cfe-421e-81e1-94a1d744582e)
https://github.com/minio/docs
my machine consumes all available memory. It's 8 cores, 48 GB virtual machine. I saw similar problem here and changed max-old-space-size in .env file toNODE_OPTIONS="--max-old-space-size=8192";
but nothing changed. I tried even bigger max-old-space-size with same result. I did succeed to add smaller repos 2-3 MB unpacked in upload folder, but anything bigger then 15 MB crashes machine.