Local Docs indexing crashes, work done disappears and it no longer allows addition of collections.

[mintradesign]

Bug Report

When Adding a large folder to local docs it will process that folder for some time succesfully then the gpt4all hangs. On a restart the document collect ion has gone. It will also not allow the addition of new document colection.

Steps to Reproduce

1. Select Add a collection then select the location (local or network connected) and let it index the selection for a few hours.
2. The system crashes or reboots (not responding)
3. Restart gpt4all and now the pre indexed local docs has gone and it will not let you add another one even after full reboot.

Expected Behavior

It should retain the indexed data for use next time GPT4all is started or continue to index/embed.

Environment

• GPT4All version: 3.1
• Operating System: Mac Mini m2 Sonoma 14.42 also happens on Threadripper with Windows 11 (Am trying on Windows Server 2022 on Epyc 12 core and it has not crashed yet.

Data size is 1.4tb but not all of it is in an indexable format as it also contains word excel and powerpoint which I would like to include but is not in the supported format list.

• Chat model used (if applicable): N/A but trying the Llama 3.1 instruct.

[mintradesign]

Just noticed that the Mac when I left GPT4All running whilst not allowing me to add a collection after some time it started to index and then is embedding the original collection. This suggests possibly that with large collections something is going on in the background in a restart, there is no visible indication that this is happeneing. I will try the PC but this is more prone to a crash after GPT4All is started.

[AndriyMulyar]

You're indexing 1.4 TB of data with LocalDocs? How big is your hard drive

[mintradesign]

Andriy It is 4Tb where the files are located, the one that seems to be working now is the M" Mac which has a 2TB drive with 500Gb free and about 2/4 of the index embedding done. I will try on a Linux PC with a 24Gb Tesla card.