Embedding process become slow when editing files or while embedding several LocalDocs folder at one time

[lauhub]

Bug Report

Embedding becomes slow after a modification of one of the files contained in LocalDocs folder

Steps to Reproduce

1. Embed a first local folder in LocalDocs (e.g. /home/user/a) that contains Markdown files
2. Wait for end of process (it is fast actually)
3. Embed a second local folder in LocalDocs (e.g. /home/user/b)
4. Without waiting for end of step 3, edit one of the Markdown files (in an external editor) that is in /home/user/a subtree (in an external editor)
5. Embedding process becomes very slow on both trees /home/user/a and /home/user/b and takes hours

Expected Behavior

Embedding process after step 4 should be as fast as in step 2

Your Environment

• GPT4All version (if applicable): 3.4.2
• Operating System: macOS 14.7 (ARM M3)
• Chat model used (if applicable): Llama 3 8B instruct

Additional notes

Same problem occurs when trying to embed 3 entries at the same time

It seems that chat process takes 100% of each CPU core during embedding process. Quitting application takes a long time (showing "Force quit" option for several minutes).

Trying to remove one of the LocalDocs entry and then quitting application (after waiting) seems to solve the problem when running again chat4all (but without adding a second LocalDocs entry).

It seems that processing several LocalDocs entries at the same time has the same effect.