snexus
commented
1 day ago Think it is more about the total size rather than number of documents. I've used the package on hundreds of megabytes of pdfs / markdowns, it might just take more time.
You can choose to embed the documents incrementally, by added few thousand documents to the folder each time, Let me know if you face any problems...
I have potentially 10s of thousands of documents. Some are as small as a few sentences while others are transcriptions of 1 hour lectures. Is there a limit to how many documents this could support? Or does it just depends on my hardware?