search

n4ze3m / dialoqbase

Create chatbots with ease
https://dialoqbase.n4ze3m.com/
MIT License
1.63k stars 266 forks source link

Adding Git Repo Issues #259

Open ghost opened 6 months ago

ghost commented 6 months ago

The Git Repo as a data source feature is almost useless. Even with 12 gigs of ram dedicated to dialoq is fails to ingest most repos. I had an idea that may be more efficient and easy to implement. There are quite a few report to pdf tools that are fast an use almost no ram. Example: https://github.com/BankkRoll/repo2pdf this clones a repo, and can convert each file into a PDF. If you used a tool like this in the background then chunk and ingest the PDFs it may actually be more efficient.

n4ze3m commented 6 months ago

Thank you. Usually, repos with large file sizes cause memory issues. We will fix it in the upcoming updates