OSCAR 22.XX scope - Githubissues

Uinelj commented 2 years ago

This issue serves as a discussion/checklist elaboration for the next OSCAR version to come.

We shoud aim to fix existing bugs/problems as well as adding potential features.

Issues

[ ] Increase robustness of newline handling in documents. (Ensure that we don't have problems storing newlines, newline characters and so on, using Rust and Python so that end users don't have issues) (see #18 )
[ ] Enforce correct language tagging. We should have something that is perfectly BCP-47 valid with no mismatches (als/gsw) (see https://github.com/oscar-corpus/ungoliant/issues/53). Ideally, add a verification layer after sentence identification to correct potentially erroneous tags, and report places where such translations couldn't have been made.
[ ] Systematically inspect very low resourced languages and remove subcorpora where data is not usable (https://github.com/oscar-corpus/corpus/issues/5, https://github.com/oscar-corpus/corpus/issues/6, https://github.com/oscar-corpus/corpus/issues/10, https://github.com/oscar-corpus/corpus/issues/11; https://github.com/oscar-corpus/corpus/issues/12, https://github.com/oscar-corpus/corpus/issues/13, https://github.com/oscar-corpus/corpus/issues/14)

[x] Add other blocklists (from UT1). Settle on which ones
[x] Add KenLM model based filtering.
[ ] Rework the annotation part: adult is too strong for something we know has a lot of false positives. Also, with the inclusion of model based filtering, we'll have to find a way to specify annotation source.

Uinelj commented 2 years ago

[x] zstd instead of gzip. Check how we can use dictionnaries. Check if we can do multipart?

ruaMIsmail commented 2 years ago

We have the sampling already finished? no

Uinelj commented 2 years ago

Sampling is related to oscar-tools and has already been merged: https://github.com/oscar-corpus/oscar-tools/pull/23

This issue tracks the changes and new features of the output corpus