issues
search
bitextor
/
warc2text
Extracts plain text, language identification and more metadata from WARC records
MIT License
20
stars
5
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
EasyBuild configs and installation instructions in the README
#60
nvanva
closed
1 month ago
0
Recover after errors opening one of WARC files in a batch
#59
nvanva
opened
3 months ago
1
Add the detected encoding to the metainformation
#58
nvanva
opened
4 months ago
0
Replacing boost::json with nlohmann::json
#57
ZJaume
closed
4 months ago
0
Filter records by HTTP status code
#56
ZJaume
opened
5 months ago
0
Fail when WARC file, tagfilters or urlfilters can't be opened
#55
ZJaume
closed
5 months ago
0
segmentation fault in case an input file doesn't exist
#54
nvanva
closed
5 months ago
0
all urls are filtered in case the file specified in --url-filters doesn't exist
#53
nvanva
closed
5 months ago
0
Move JSONL output to --stdout and allow file-based output with JSONL
#52
ZJaume
closed
7 months ago
0
ZSTD compression and compression level support
#51
ZJaume
closed
7 months ago
0
warc2html changes
#50
ZJaume
closed
7 months ago
1
Domain filtering with a user-defined list, non regex based
#49
ZJaume
opened
8 months ago
1
toUTF8 skips characters instead of throwing on encoding issues
#48
kpu
closed
7 months ago
0
GitHub CI
#47
jelmervdl
opened
10 months ago
1
Track html tags
#46
jelmervdl
opened
10 months ago
1
Combining `--multilang` and paragraph-level annotations
#45
jelmervdl
opened
10 months ago
0
Crash with `--paragraph-identification`.
#44
jelmervdl
closed
10 months ago
2
Add `--robotspass` shunt for records related to robots.txt
#43
jelmervdl
closed
10 months ago
5
Lowercase before langid
#42
ZJaume
opened
11 months ago
2
Shunt robots.txt responses to separate warc
#41
jelmervdl
closed
10 months ago
0
[META] Summary of HPLT requested changes
#40
jelmervdl
opened
11 months ago
0
Compilation error (fasttext)
#39
cgr71ii
opened
1 year ago
5
Release versions
#38
jelmervdl
closed
1 year ago
10
Split logging between stderr and stdout
#37
nvanva
opened
1 year ago
1
Add fasttext as an option alongside cld2
#36
jelmervdl
closed
1 year ago
12
Add `--jsonl` option
#35
jelmervdl
closed
7 months ago
10
alternative output format based on JSONlines
#34
oepen
closed
7 months ago
5
Paragraph identification
#33
cgr71ii
closed
2 years ago
0
xh_scanner loses data when tag name or attribute is too long
#32
jelmervdl
opened
2 years ago
0
Don't parse text/plain as HTML
#31
jelmervdl
closed
3 years ago
1
Respect the original casing of the WARC-Target-URI field
#30
zuny26
closed
3 years ago
0
Improve compilation
#29
zuny26
closed
3 years ago
0
Optimize regex filters
#28
jelmervdl
closed
3 years ago
0
Mark url filter notices as level "info"
#27
jelmervdl
closed
3 years ago
0
Add --encode-urls option to deal with unescaped chars in URLs
#26
zuny26
closed
3 years ago
1
Add url filter, similar to the tag filter we use for MT
#25
jelmervdl
closed
3 years ago
0
What happened to Hausa?
#24
kpu
closed
3 years ago
3
Use CLD2 extended detection method to detect multiple languages in a single document
#23
zuny26
closed
3 years ago
0
Update preprocess
#22
jelmervdl
closed
3 years ago
0
Ignore zip errors
#21
jelmervdl
closed
3 years ago
0
warc2text tries to read large warc records into memory, causing OOM
#20
jelmervdl
opened
3 years ago
1
Remove work-around for StringPiece confusion.
#19
jelmervdl
closed
3 years ago
0
Replace zip extraction
#18
jelmervdl
closed
3 years ago
1
Skip large pdfs
#17
jelmervdl
closed
3 years ago
0
assertion failure in compress.cc
#16
jelmervdl
closed
3 years ago
7
Crash with std::invalid_argument on stoul call
#15
jelmervdl
closed
3 years ago
3
Fix compilation issue with gcc.
#14
jelmervdl
closed
3 years ago
0
Allow comments in tag filter file
#13
jelmervdl
closed
3 years ago
0
New options
#12
zuny26
closed
3 years ago
0
Open document format support
#11
lpla
closed
3 years ago
0
Next