issues
search
allenai
/
bff
Apache License 2.0
37
stars
8
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Modified BFF version mentioned in DataComp-LM paper
#13
SinclairCoder
opened
2 months ago
0
Big OpenLM/DCLM <-> AI2 PR # 1
#12
revbucket
opened
4 months ago
0
Added i) directory support, ii) FP rate args, iii) No-Save option
#11
revbucket
opened
5 months ago
1
Barebones python/S3 implementation
#10
revbucket
opened
6 months ago
0
Is the deduplication scope separate or global when deduplicating multiple files?
#9
RulinShao
closed
3 months ago
1
lint fixes
#8
chris-ha458
closed
11 months ago
0
Backport improvements from the Dolma repo
#7
dirkgr
opened
11 months ago
1
add option to always hash whole paragraphs
#6
IanMagnusson
closed
1 year ago
0
bff_duplicate_spans are index in bytes not characters
#5
IanMagnusson
opened
1 year ago
0
Adds the option to run on a whole document at a time
#4
dirkgr
closed
1 year ago
1
Ngram instead of paragraph removal?
#3
IanMagnusson
opened
1 year ago
1
Output to a file instead of stdout
#2
dirkgr
closed
11 months ago
1
Code should use threads to run in parallel
#1
dirkgr
closed
1 year ago
1