search

marian-nmt / marian-examples

Examples, tutorials and use cases for Marian, including our WMT-2017/18 baselines.
Other
78 stars 34 forks source link

Add transformer basic example #26

Closed graemenail closed 2 years ago

graemenail commented 2 years ago

Adds a basic transformer example using SentencePiece.

The model uses the transformer-base task as a starting point, and adjusts the configuration to validate and save more frequently for demonstration. It uses a subset of the WMT21 en-de news-task data, with WMT19 as a validation set and WMT20 as a test set. Data cleaning is minimal, with non-printing characters removed, length constraint of 1-100 words, and deduplication. The data consists of EuroParl, news-commentary and commoncrawl.

It trains for 10 epoch, or 10 stalled validations on ce-mean-word, whichever occurs first. It validates on ce-mean-word, bleu and perplexity. This took around 8hr on 2 RTX 2080s with a workspace of 9000MiB.

The results of the model are shown for WMT16-20, for BLEU, chrF2 and Comet (wmt20-comet-da).

There is a comet-script provided for convenience as well. It is written such that it can also be used as a validation script for Marian: i.e. ./comet-score /tmp/marian-outut.txt is scored against the validation set. I found that when forced to execute on the CPU, it was a little slow for this example.

graemenail commented 2 years ago

Thanks for the comments @snukky ; I've addressed them all.

Merging