search

bitextor / bicleaner

Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.
GNU General Public License v3.0
148 stars 22 forks source link

readme

bicleaner

License

Bicleaner (bicleaner-classify) is a tool in Python that aims at detecting noisy sentence pairs in a parallel corpus. It indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0). Sentence pairs considered very noisy are scored with 0.

Although a training tool (bicleaner-train) is provided, you may want to use the available ready-to-use language packages. Please, visit https://github.com/bitextor/bicleaner-data/releases/latest or use bicleaner-download to download the latest language packages. Visit our Wiki for a detailed example on Bicleaner training.

Citation

If you find Bicleaner useful, please consider citing the following papers:

V. M. Sánchez-Cartagena, M. Bañón, S. Ortiz-Rojas and G. Ramírez-Sánchez,\ "Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task",\ in Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers.\ Brussels, Belgium: Association for Computational Linguistics, October 2018

@InProceedings{prompsit:2018:WMT,
  author    = { V\'{i}ctor M. S\'{a}nchez-Cartagena and Marta Ba{\~n}\'{o}n and Sergio Ortiz-Rojas and Gema Ram\'{i}rez-S\'{a}nchez},
  title     = {Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task},
  booktitle = {Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers},
  month     = {October},
  address   = {Brussels, Belgium},
  publisher = {Association for Computational Linguistics}
}

Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón and Sergio Ortiz Rojas \ "Bifixer and Bicleaner: two open-source tools to clean your parallel data.",\ in Proceedings of the 22nd Annual Conference of the European Association for Machine Translation.\ Lisboa, Portugal: European Association for Machine Translation, November 2020

@InProceedings{prompsit:2020:EAMT,
  author    = {Gema Ram\'{i}rez-S\'{a}nchez and Jaume Zaragoza-Bernabeu and Marta Ba{\~n}\'{o}n and Sergio Ortiz-Rojas},
  title     = {Bifixer and Bicleaner: two open-source tools to clean your parallel data.},
  booktitle = {Proceedings of the 22nd Annual Conference of the European Association for Machine Translation},
  pages     = {291--298},
  isbn      = {978-989-33-0589-8},
  year      = {2020},
  month     = {November},
  address   = {Lisboa, Portugal},
  publisher = {European Association for Machine Translation}
}

Installation & Requirements

Bicleaner is written in Python and can be installed using pip. It also requires the KenLM Python bindings with support for 7-gram language models. You can easily install it by running the following commands:

pip install bicleaner
pip install --config-settings="--build-option=--max_order=7" https://github.com/kpu/kenlm/archive/master.zip

The remaining extra modules required by Bicleaner will be automatically downloaded and installed/upgraded (if required) with the first command.

Also, you can install the conda package (KenLM is already included):

conda install -c conda-forge -c bitextor bicleaner

After installation, three binary files (bicleaner-train, bicleaner-classify and bicleaner-classify-lite) will be located in your python/installation/prefix/bin directory. This is usually $HOME/.local/bin or /usr/local/bin/.

Cleaning

bicleaner-classify aims at detecting noisy sentence pairs in a parallel corpus. It indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0). Sentence pairs considered very noisy are scored with 0.

By default, the input file (the parallel corpus to be classified) must contain at least four columns, being:

but the source and target sentences column index can be customized by using the --scol and --tcol flags.

The generated output file will contain the same lines and columns that the original input file had, adding an extra column containing the Bicleaner classifier score.

This tool can be run with

bicleaner-classify [-h]
                   [-S SOURCE_TOKENIZER_COMMAND]
                   [-T TARGET_TOKENIZER_COMMAND] 
                   [--header]
                   [--scol SCOL]
                   [--tcol TCOL] 
                   [--tmp_dir TMP_DIR]
                   [-b BLOCK_SIZE] 
                   [-p PROCESSES] 
                   [-d DISCARDED_TUS]
                   [--lm_threshold LM_THRESHOLD] 
                   [--score_only]
                   [--disable_hardrules]
                   [--disable_lm_filter]
                   [--disable_porn_removal]
                   [--disable_minimal_length]
                   [-q] 
                   [--debug] 
                   [--logfile LOGFILE] 
                   [-v]
                   input 
                   [output] 
                   metadata

Parameters

Example

bicleaner-classify  \
        corpus.en-es.raw  \
        corpus.en-es.classifed  \
        training.en-es.yaml

This will read the "corpus.en-es.raw" file, classify it with the classifier indicated in the "training.en-es.yaml" metadata file, writing the result of the classification in the "corpus.en-es.classified" file. Each line of the new file will contain the same content as the input file, adding a column with the score given by the Bicleaner classifier.

Automatic test

We included a small test corpus and a script to check that your Bicleaner classifier is working as expected. In order to use it, just run:

python -m pytest -s tests/bicleaner_test.py

This will download the required language pack, classify the provided test corpus, and check the resulting classification scores. If everything went as expected, the output will be "1 passed in XX.XX seconds". All downloaded data will be removed at the end of the testing session.

Training classifiers

In case you need to train a new classifier (i.e. because it is not available in the language packs provided at bicleaner-data), you can use bicleaner-train . bicleaner-train is a Python3 tool that allows you to train a classifier which predicts whether a pair of sentences are mutual translations or not and discards too noisy sentence pairs. Visit our Wiki for a detailed example on Bicleaner training.

Requirements

In order to train a new classifier, you must provide:

$ cat monolingual.SL \
    | sacremoses -l SL tokenize -x \
    | awk '{print tolower($0)}' \
    | tr ' ' '\n' \
    | LC_ALL=C sort | uniq -c \
    | LC_ALL=C sort -nr \
    | grep -v '[[:space:]]*1' \
    | gzip > wordfreq-SL.gz
$ cat monolingual.TL \
    | sacremoses -l TL tokenize -x \
    | awk '{print tolower($0)}' \
    | tr ' ' '\n' \
    | LC_ALL=C sort | uniq -c \
    | LC_ALL=C sort -nr \
    | grep -v '[[:space:]]*1' \
    | gzip > wordfreq-TL.gz

Optionally, if you want the classifier to include a porn filter, you must also provide:

Optionally, if you want the classifier to include an improved fluency filter based on language models, you must also provide:

If not provided, since Bicleaner 0.13, noisy corpora is produced synthetically from the training corpus.

Moreover, lmplz, the command to train a KenLM language model must be in PATH. See https://github.com/kpu/kenlm for instructions about its compilation and installation.

In principle, if you want to use Bicleaner to clean a partially noisy corpus, it could be difficult to find a corpus made solely of noisy sentences. Fortunately, there are two options available with Bicleaner:

Extracting noisy sentences from an existing corpus with heuristic rules

Given a parallel corpus, you use bicleaner-hardrules to extract some of its noisiest sentences using heuristic rules by running the following command:

  bicleaner-hardrules [-h]
                      [--annotated_output]
                      -s SOURCE_LANG 
                      -t TARGET_LANG
                      [--tmp_dir TMP_DIR]
                      [-b BLOCK_SIZE]
                      [-p PROCESSES]
                      [--disable_lang_ident]
                      [--disable_minimal_length]
                      [--header]
                      [--scol SCOL]
                      [--tcol TCOL]
                      [--disable_lm_filter] 
                      [--disable_porn_removal]
                      [--metadata METADATA]
                      [--lm_threshold LM_THRESHOLD]
                      [-q] 
                      [--debug]
                      [--logfile LOGFILE]
                      [input]
                      [output]

where INPUT_FILE contains a sentence-aligned parallel corpus, with a sentence pair per line. Sentences are split by tab. OUTPUT_FILE will contain all the input sentences, with an extra score column with 0 (if the sentence is noisy and should be discarded) or 1 (if the sentence is ok). When the --annotated_output flag is in use, OUTPUT_FILE will contain another extra column, specifying the heuristic rule applied to decide discarding each sentence (or keep, if the sentence is ok and should not be discarded). If the --disable_lang_ident flag is in use, rules that require language identification are not used. '--scol' and '--tcol' allow to indicate which columns contains source and target in the input file (default: 1and 2, respectively).

In order to use the LM filtering and/or porn removal, you must provide the --metadata (it is: the .yaml file generated by Bicleaner training). To disable LM filtering and/or porn removal, just use the --disable_lm_filter and/or --disable_porn_removal flags.

You can then obtain the monolingual noisy corpora by "cutting" the appropriate columns (after running bicleaner-hardrules with the --annotated_output flag). Asuming scol=1 and tcol=2, and no more columns in the input corpus (so the hardrules score is the 3rd column in the output):

cat OUTPUT_FILE | awk -F'\t' '{if ($3 == 0) print $1 }' > MONOLINGUAL_NOISY.SOURCE_LANG
cat OUTPUT_FILE | awk -F'\t' '{if ($3 == 0) print $2 }' > MONOLINGUAL_NOISY.TARGET_LANG

Building synthetic noisy sentences

cat TRAINING_CORPUS | cut -f1 | python3.7 bicleaner/utils/shuffle.py - > MONOLINGUAL_NOISY.SOURCE_LANG
cat TRAINING_CORPUS | cut -f2 | python3.7 bicleaner/utils/shuffle.py - > MONOLINGUAL_NOISY.TARGET_LANG

Since 0.13, if no noisy corpora is provided, it's produced by Bicleaner training itself, so it has become an optional parameter.

Parameters

It can be used as follows. Note that the parameters --noisy_examples_file_sl, --noisy_examples_file_tl, --lm_file_sl, --lm_file_tl, are mandatory if you want to enable improved fluency filter based on language models (recommended).

bicleaner_train.py [-h]
    -m METADATA
    -c CLASSIFIER
    -s SOURCE_LANG
    -t TARGET_LANG
    -d SOURCE_DICTIONARY
    -D TARGET_DICTIONARY
    -f SOURCE_WORD_FREQS
    -F TARGET_WORD_FREQS
    [-S SOURCE_TOKENIZER_COMMAND]
    [-T TARGET_TOKENIZER_COMMAND]
    [--normalize_by_length]
    [--treat_oovs]
    [--qmax_limit QMAX_LIMIT]
    [--disable_features_quest]
    [--classifier_type {mlp,svm,nn,nn1,adaboost,random_forest,extra_trees}]
    [--dump_features DUMP_FEATURES]
    [-b BLOCK_SIZE]
    [-p PROCESSES]
    [--wrong_examples_file WRONG_EXAMPLES_FILE]
    [--features_version FEATURES_VERSION]
    [--disable_lang_ident]
    [--seed SEED]
    [--relative_paths]
    [--noisy_examples_file_sl NOISY_EXAMPLES_FILE_SL]
    [--noisy_examples_file_tl NOISY_EXAMPLES_FILE_TL]
    [--lm_dev_size LM_DEV_SIZE]
    [--lm_file_sl LM_FILE_SL]
    [--lm_file_tl LM_FILE_TL]
    [--lm_training_file_sl LM_TRAINING_FILE_SL]
    [--lm_training_file_tl LM_TRAINING_FILE_TL]
    [--lm_clean_examples_file_sl LM_CLEAN_EXAMPLES_FILE_SL]
    [--lm_clean_examples_file_tl LM_CLEAN_EXAMPLES_FILE_TL]
    [--porn_removal_train PORN_REMOVAL_TRAIN]
    [--porn_removal_test PORN_REMOVAL_TEST]
    [--porn_removal_file PORN_REMOVAL_FILE]
    [--porn_removal_side {sl,tl}]
    [-q] [--debug] [--logfile LOGFILE]
    [input]

Example

bicleaner-train \
          corpus.en-cs.train\
          --normalize_by_length \
          -s en \
          -t cs \
          -d dict-en-cs.gz \
          -D dict-cs-en.gz \
          -f wordfreqs-en.gz \
          -F wordfreqs-cs.gz \
          -c en-cs.classifier \
          --lm_training_file_sl lmtrain.en-cs.en --lm_training_file_tl lmtrain.en-cs.cs \
          --lm_file_sl model.en-cs.en  --lm_file_tl model.en-cs.cs \
          --porn_removal_train porn-removal.txt.en  --porn_removal_file porn-model.en \
          -m training.en-cs.yaml \

This will train an Extra Trees classifier for English-Czech using the corpus corpus.en-cs.train, the probabilistic dictionaries dict-en-cs.gz and dict-cs-en.gz, and the word frequency dictionaries wordfreqs-en.gz and wordfreqs-cs.gz. This training will use 50000 good and 50000 bad examples. The classifier data will be stored in en-cs.classifier, with the metadata in training.en-cs.yaml. The improved fluency language models will be model.en-cs.en and model.en-cs.cs, and the porn filter model will be porn-model.en.

The generated .yaml file provides the following information, that is useful to get a sense on how good or bad was the training (and is also a needed input file for classifying):

classifier: en-cs.classifier
classifier_type: extra_trees
source_lang: en
target_lang: cs
source_dictionary: dict-en-cs.gz
target_dictionary: dict-cs-en.gz
source_word_freqs: wordfreqs-en.gz
target_word_freqs: wordfreqs-cs.gz
normalize_by_length: True
qmax_limit: 40
disable_features_quest: True
good_test_histogram: [0, 7, 39, 45, 112, 172, 514, 2199, 6912, 0]
wrong_test_histogram: [14, 4548, 4551, 747, 118, 18, 3, 1, 0, 0]
precision_histogram: [0.5000000, 0.5003502, 0.6475925, 0.9181810, 0.9860683, 0.9977594, 0.9995846, 0.9998903, 1.0000000, nan]
recall_histogram: [1.0000000, 1.0000000, 0.9993000, 0.9954000, 0.9909000, 0.9797000, 0.9625000, 0.9111000, 0.6912000, 0.0000000]
accuracy_histogram: [0.5000000, 0.5007000, 0.7277500, 0.9533500, 0.9884500, 0.9887500, 0.9810500, 0.9555000, 0.8456000, 0.5000000]
length_ratio: 1.0111087
features_version: 4
source_lm: model.en-cs.en
target_lm: model.en-cs.cs
lm_type: CHARACTER
clean_mean_perp: -1.0744755342473238
clean_stddev_perp: 0.18368996884800565
noisy_mean_perp: -3.655791900929066
noisy_stddev_perp: 0.9989343799121657
disable_lang_ident: False
porn_removal_file: porn-model.en
porn_removal_side: sl

Lite version

Although bicleaner-train and bicleaner-classify make use of parallelization by distributing workload to the available cores, some users might prefer to implement their own parallelization strategies. For that reason, single-thread version of Bicleaner classifier script is provided: bicleaner-classify-lite. The usage is exactly the same as for the full version, but omitting the blocksize (-b) and processes (-p) parameter. Note: bicleaner-train-lite was removed due to the lack of usage by the users and to avoid code duplication.

Connecting Europe Facility

All documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.