johnbrisbin
commented
1 year ago You are right. As a 'excessive threader' you have been punished.
There is an inconsiderate choice in the ingestion code which tries to launch as many processes to ingest from files as there are threads proclaimed by your processor (64, in your case). Winders doesn't like that many processes being launched all at once and errors out. It was a very poor idea, anyway.
The offending code in ingest.py is this:
with Pool(processes=os.cpu_count()) as pool:
if you change it to something more reasonable like:
with Pool(processes=min(8, os.cpu_count())) as pool:
I know this because it will blow up with even 32 threads like mine.
You will get quite good performance without thrashing the disk too much with a maximum of 8 processes launched (though still not ideal). Best if the data is on an SSD, of course.
This remains a bug until that bit is rewritten for everyone else.
adontcare
Describe the bug and how to reproduce it I have performed all of the steps for setting up PrivateGPT, and I'm confident I have not made any mistakes.
When running ingest.py for the first time with only the default .txt file within source_documents, I am hit with the following error which may be related to some incompatibility with my AMD Threadripper CPU maybe having too many cores that PrivateGPT maybe doesn't know what to do with them all:
Here is the error:
PS C:\Users\Microshit\privateGPT> & C:/Users/Microshit/AppData/Local/Programs/Python/Python311/python.exe c:/Users/Microshit/privateGPT/ingest.py Creating new vectorstore Loading documents from source_documents Exception in thread Thread-1 (_handle_workers): Traceback (most recent call last): File "C:\Users\Microshit\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner self.run() File "C:\Users\Microshit\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run Loading new documents: 0%| | 0/1 [00:00<?, ?it/s] self._target(*self._args, **self._kwargs) File "C:\Users\Microshit\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 522, in _handle_workers cls._wait_for_updates(current_sentinels, change_notifier) File "C:\Users\Microshit\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 502, in _wait_for_updates wait(sentinels, timeout=timeout) File "C:\Users\Microshit\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\connection.py", line 878, in wait ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Microshit\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\connection.py", line 810, in _exhaustive_wait res = _winapi.WaitForMultipleObjects(L, False, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: need at most 63 handles, got a sequence of length 66
Expected behaviour Whilst running ingest.py on state_of_blah.txt, I expected no errors to be seen.
Environment (please complete the following information):