Open andypbaker opened 1 month ago
I have been facing this issue. The file name is clean and only contains upper-case, lower-case, numbers, and hyphen, so it is not duplicate of #2658 .
I went through a long long path of finding what has caused the crash. Surprisingly enough, the problematic file is NOT the one that is shown in gpt4all GUI (number 2). I found this by copy-pasting txt files one-by-one to that folder and see how the embedding process goes.
I finally found the file and manage to get it fixed by removing all the "weird" characters
sed -i 's/[^a-zA-Z0-9 \-\(\)\.,;:\>\<\?=+%]//g' THE-FILE
So next step was to find which character was the problematic one. This short R script shows us the way:
library("Unicode")
setwd("/home/mehrad/Documents/Data/gpt4all/test/")
## read files
dirty <- readLines("dirty.txt", warn = F)
cleaned <- readLines("cleaned.txt", warn = F)
## split to single unique characters
cleaned_chars <- strsplit(x = cleaned, split = "") |>
unlist() |>
unique()
dirty_chars <- strsplit(x = dirty, split = "") |>
unlist() |>
unique()
# get the diff of these chars along with their unicode representation
setdiff(dirty_chars, cleaned_chars) |>
sapply(function(x){
tmp_unicode <- as.character(Unicode::as.u_char(utf8ToInt(x)))
return(c(x, tmp_unicode))
}) |>
rbind.data.frame() |>
t()
This results in
[,1] [,2] / "/" "U+002F" - "-" "U+002D" Š "Š" "U+0160" — "—" "U+2014" \002 "\002" "U+0002" “ "“" "U+201C" ” "”" "U+201D" ’ "’" "U+2019" α "α" "U+03B1" | "|" "U+007C" ∂ "∂" "U+2202" − "−" "U+2212" _ "_" "U+005F" ≥ "≥" "U+2265" Δ "Δ" "U+0394" ∪ "∪" "U+222A" Π "Π" "U+03A0" ∈ "∈" "U+2208" × "×" "U+00D7" β "β" "U+03B2" γ "γ" "U+03B3" { "{" "U+007B" } "}" "U+007D" ′ "′" "U+2032" ‖ "‖" "U+2016" ≤ "≤" "U+2264" λ "λ" "U+03BB" η "η" "U+03B7" ˜ "˜" "U+02DC" ˆ "ˆ" "U+02C6" √ "√" "U+221A" ï "ï" "U+00EF"
one of these characters is breaking gpt4all . I tried to insert each character in a text file (in hello {} world
the character would have replaced {}
) and make the gpt4all to "embed" them. This attempt failed which shows that whatever it is related to these character (since removing them from the text fixed the issue), but how it affect the indexer is unknown to me.
I also have same exact problem, probably an issue with turkish characters.
Bug Report
updated to version 3.1.0 localdocs collections had to be updated. selected update (can't remember what the verb was but it meant update). my colection folder contained 47 documents of varying sizes. embedding got to 93% after about 8 hrs. then the % chaneg to 0% and the number of embeddings of total embeddings changed to -18446744073709319000 of 33026 embeddings. it might have got to 32767 then turned negative.![localdocs didn't cope, got to 93 percent then went negative Screenshot 2024-07-24 185412](https://github.com/user-attachments/assets/d86d641e-145b-4c89-88a7-53e8ee6c74ff)
Steps to Reproduce
Expected Behavior
expected it to reach 100% complete.
Your Environment