Next steps sentence alignment

simon-clematide commented 6 years ago

[x] @siclemat clean up repo
[x] tokenisierung ; spacy oder cutter
[x] document translation (de => fr)
[x] monolingual sentence alignment with bleu-champ

aflueckiger commented 6 years ago

Alignments:

[x] segment and tokenize all docs (cutter)
[x] either create xml or txt to feed that is required for Chantal's scripts
[x] use Chantal's script that aligns documents with BLEUalignment
[x] train MT model with data from bilingwisslc and JRC-Acquis
[x] update makefile to cover entire pipeline
[x] evaluate document alignment for a single year per decade (2010, 2000, 1990)
[x] retrain MT model with domain-specific data if necessary
[x] monolingual sentence alignment with bleu-champ

Scraping:

[ ] record date of download in tsv
[x] record relative path to each document in tsv

simon-clematide commented 5 years ago

The metadata on pages is not accurate for "Beilagen" or "Bereichte". They show a page length of 1 (e.g. 174-174) although the PDF contains several pages (but they do not follow the page numbering of the volume). We need to add the physical number of pages to the statistics file e.g. article-info-de.tsv

aflueckiger commented 5 years ago

The physical page numbers are included in the file article-info-XX.tsv

impresso / federal-gazette

Next steps sentence alignment #3