issues
search
google-research
/
deduplicate-text-datasets
Apache License 2.0
1.09k
stars
108
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Bump tensorflow from 2.9.0 to 2.12.1
#52
dependabot[bot]
opened
1 month ago
0
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
#51
bingkunyao
opened
2 months ago
1
Format rust code via `cargo fmt`
#50
nathan-barry
opened
3 months ago
1
Distributed running
#49
jordane95
opened
3 months ago
0
does finish_dedup_wiki40b.py has some wrong?
#48
mathCrazyy
opened
3 months ago
0
does this tool can process Chinese?
#47
mathCrazyy
opened
3 months ago
1
Fix issue #45:out of range bug
#46
WWWonderer
closed
3 months ago
5
[Bug] Out of range error when counting occurrences on a custom suffix array
#45
WWWonderer
closed
3 months ago
1
Question: Upper Bound
#44
bezir
closed
4 months ago
1
Count_occurrence does not work with tokenizer?
#43
WWWonderer
closed
4 months ago
2
是否可以提供一个纯python版本的，相信很多研究者在服务器上没有权限安装gcc
#42
gongye19
closed
4 months ago
1
question about wstring_equal function
#41
WWWonderer
closed
4 months ago
2
where the data is?
#40
jianshu93
closed
5 months ago
1
when i use tokenizer , I obtained many patterns that span across the data, which is quite strange.
#39
gawei1995
opened
6 months ago
4
customized dataset deduplication
#38
zengyangjie
closed
5 months ago
1
Bump tensorflow from 2.9.0 to 2.11.1
#37
dependabot[bot]
closed
1 month ago
1
[Question] An error with the same repo guideline
#36
manel-hikk
opened
7 months ago
5
remove_ex in finish_dedup_wiki40b
#35
wead-hsu
closed
6 months ago
1
Incomplete Sentences
#34
MiladMolazadeh
closed
6 months ago
1
Adjust TensorFlow version to fix cuDNN, cuFFT, cuBLAS errors.
#33
longxudou
closed
6 months ago
4
Retain one instance per duplicate
#32
RobinQrtz
closed
9 months ago
2
what is the input dataset format for custom dataset?
#31
mittsommer
closed
6 months ago
1
cargo build error.Could you upload cargo.lock file?
#30
jinzhuer
opened
1 year ago
2
How to restore the result data after deduplication (remove invisible characters)
#29
greenriver777
closed
6 months ago
1
[Paper Question] Why use w-shingles over k-shingles?
#28
micimize
closed
1 year ago
1
Inplementation of NearDup(approximate match)
#27
Yaoming95
closed
1 year ago
1
Simple test
#26
KeremTurgutlu
closed
1 year ago
0
[Proposal] Use `release` build, not `debug` builds
#25
Narsil
closed
6 months ago
3
Off-by-1 error in `collect`?
#24
ola13
opened
1 year ago
0
question about deduplication cluster size
#23
everks
closed
1 year ago
2
make cmd_merge use multiple threads again
#22
TristanThrush
closed
6 months ago
1
how to deduplicate huggingface datasets
#21
StephennFernandes
closed
6 months ago
7
Accessing the duplicates and their counts
#20
yanaiela
closed
1 year ago
13
Fix to issue #17 limits cmd_merge to be single-threaded
#19
kleinj
closed
6 months ago
3
RAM crash when use collect method
#18
acul3
opened
2 years ago
2
one bug when I use
#17
flyingwaters
closed
2 years ago
2
Should newline char be removed
#16
cperiz
closed
2 years ago
1
Unexpected behavior with ending symbols
#15
mitya52
closed
2 years ago
2
"failed to fill whole buffer" errors
#14
mitya52
closed
2 years ago
2
Fix multiprocessing bug in Windows/Mac OS X
#13
alistairewj
closed
2 years ago
1
Error when running the code
#12
MatthewCYM
closed
6 months ago
15
Error with table size not being divisible by text size
#11
jinyongyoo
closed
2 years ago
7
Dev v1
#10
carlini
closed
2 years ago
1
false positives
#9
ChenghaoMou
closed
2 years ago
1
Can the tool run on plain text files?
#8
m-resta
closed
2 years ago
20
How to dedup subtring in one dataset?
#7
lan2016286
closed
2 years ago
9
Adding possibility to load an HF-dataset
#6
TevenLeScao
closed
6 months ago
5
Error on self deduplication
#5
zijwang
closed
2 years ago
10
Why not use Simhash?
#4
Ethan-yt
closed
3 years ago
3
How to dedup between two datasets?
#3
mralexis1
closed
2 years ago
7
Next