issues
search
allenai
/
dolma
Data and tools for generating and inspecting OLMo pre-training data.
https://allenai.github.io/dolma/
Apache License 2.0
894
stars
90
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Bump h2 from 0.3.24 to 0.3.26 in the cargo group across 1 directory
#146
dependabot[bot]
closed
4 months ago
1
Add extra tests for multi-byte unicode spans in deduper.
#145
soldni
closed
4 months ago
0
Optionally add total/sum to output of analyzer
#144
soldni
closed
5 months ago
0
S3 mixer doesn't start
#143
marcopasqua
opened
5 months ago
2
Data out of bounds when using ‘dolma tokens --dtype uint32’
#142
Jackwaterveg
opened
5 months ago
1
Add an option to improve tokenization shuffling
#141
soldni
closed
5 months ago
0
Fix local shuffling failure
#140
soldni
closed
5 months ago
0
Possible bug in `local_shuffle`?
#139
hwijeen
closed
5 months ago
2
Some race condition in url taggers
#138
peterbjorgensen
opened
5 months ago
1
use precompiled regex when loading url blocklists
#137
peterbjorgensen
closed
5 months ago
0
Is there a way to intergratge Dolma toolkit to Spark?
#136
DangoWang
opened
5 months ago
1
Improves tool to compute statistics; adds deduplication options.
#135
soldni
closed
5 months ago
0
A Question about the meaning of dolma_v1.6_cc_en
#134
aleien95
closed
5 months ago
1
Added JQ syntax for replacements + added minimum score.
#133
soldni
closed
5 months ago
0
Bump the cargo group group with 1 update
#132
dependabot[bot]
closed
5 months ago
0
Added Support for JQ syntax in include/exclude mixer config
#131
soldni
closed
6 months ago
0
Support providing streams into mixer via CLI
#130
soldni
opened
6 months ago
0
Tagger modules import (fix for #128)
#129
soldni
closed
6 months ago
0
tagger_modules do not work in current git version
#128
peterbjorgensen
closed
6 months ago
1
Can I use the dolma toolkit to process my own datasets?
#127
Tendo33
closed
6 months ago
1
Update CITATION.cff
#126
soldni
closed
6 months ago
0
make_wikipedia in getting_started.md
#125
leeparkuky
closed
4 months ago
3
Update CITATION.cff
#124
jacob-morrison
closed
6 months ago
0
not_alphanum_paragraph_v1 tagger takes forever to run on certain inputs.
#123
peterbjorgensen
opened
6 months ago
1
Option to use ngram overlap to dedupe paragraphs
#122
rodneykinney
closed
6 months ago
2
make_wikipedia.py: long running time
#121
chschroeder
opened
6 months ago
3
Add preliminary Dolma v1.7 configurations, fix corner case in tokens.
#120
soldni
closed
6 months ago
0
Add model training
#119
soldni
closed
4 months ago
0
Updated CFF and Bibtex
#118
soldni
closed
6 months ago
0
Fix issue in getting started tutorial using wikipedia data
#117
RohitRathore1
closed
5 months ago
1
Modify CI to build when version is incremented; increment to v1.0.1
#116
soldni
closed
6 months ago
0
Update README.md
#115
eltociear
closed
6 months ago
0
do not overwrite tagger outputs with the same output path, fixes #113
#114
peterbjorgensen
closed
6 months ago
1
Only the attributes written by the last tagger in the tagger list gets written in version 1.0.0
#113
peterbjorgensen
closed
6 months ago
0
Taggers for URL filtering
#112
soldni
closed
6 months ago
0
How does Exact paragraph deduplication performed?
#111
silverriver
closed
6 months ago
1
Tokenizer name or path must be found error
#110
RohitRathore1
closed
6 months ago
3
Provenance license?
#108
boxabirds
closed
6 months ago
1
Fix broken data sheet link in README
#107
simonw
closed
6 months ago
0
Data sheet link in README is broken
#106
simonw
closed
6 months ago
1
Fixing Errors in Linux Build
#105
soldni
closed
7 months ago
0
Disambiguating that the repo is for the dolma toolkit in various docs
#104
arnavic
closed
7 months ago
0
Fixed Dangling CLI Options; E2E Tokenizer Tests
#103
soldni
closed
7 months ago
0
BOS/EOS/PAD options in `tokens` cli; speed up tokenization by segmenting paragraphs.
#102
soldni
closed
7 months ago
0
Bump h2 from 0.3.20 to 0.3.24
#101
dependabot[bot]
closed
7 months ago
0
V1.0 candidate; new deduper options, new taggers
#100
soldni
closed
7 months ago
0
Fixing Build Config Issues
#99
soldni
closed
7 months ago
0
Add robust median to gopher filter
#98
KennethEnevoldsen
closed
7 months ago
7
ff
#97
soldni
closed
8 months ago
0
deduplication examples does not work
#96
TTTTao725
closed
4 months ago
5
Previous
Next