Enrich translated dataset

[stefan-it]

Hi,

it would be a great improvement, if the translated dataset can be enriched with more data or fields:

• The Original data (instruction, input and output) can be included to have a better comparison of original and translated data.
• Add support for different translation models.
• Field that is e.g. named review_needed should be added. Problematic or wrong examples can be detected (automatically or manual) and can then be flagged.

On Slack we had the discussion about markdown tables. So one could easily write a markdown table detection script and flag the found examples with the review_needed option, so that these examples can be reviewed later.

Another issue to be discussed: do we want to "override" the existing translated_german_alpaca.json? Or should we introduce a new file for that? But is more than one "dataset" confusing?

Concrete implementation

Concrete implementation steps would be to introduce the following new keys for each example in the dataset:

• instruction
• input
• output
• review_needed (Boolean, default: false).
• translations with instruction, input and output as keys

Proof of concept

One example entry of that enriched dataset could look like:

{
    "instruction": "Give three tips for staying healthy.",
    "input": "",
    "output": "1. Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule.",
    "translations": {
        "instruction": [
            {"facebook/wmt19-en-de": "Geben Sie drei Tipps, um gesund zu bleiben."},
            {"deepl/formality.less": "Gib drei Tipps, um gesund zu bleiben."}
        ],
        "input": [
            {"facebook/wmt19-en-de": ""},
            {"deepl/formality.less": ""}
        ],
        "output": [
            {"facebook/wmt19-en-de": "1. Ern\u00e4hren Sie sich ausgewogen und sorgen Sie f\u00fcr reichlich Obst und Gem\u00fcse. \n2. Sport treiben Sie regelm\u00e4\u00dfig, um Ihren K\u00f6rper aktiv und stark zu halten. \n3. Erhalten Sie gen\u00fcgend Schlaf und halten Sie einen gleichm\u00e4\u00dfigen Schlafplan."},
            {"deepl/formality.less": "1. ern\u00e4hre dich ausgewogen und achte darauf, dass du viel Obst und Gem\u00fcse isst. \n2. Treibe regelm\u00e4\u00dfig Sport, um deinen K\u00f6rper aktiv und stark zu halten. \n3. Schlafe ausreichend und halte dich an einen festen Schlafrhythmus."}
        ]
    },
    "review_needed": false
},

[PhilipMay]

For me this sounds a bit like we need a real database with GUI and dataset versioning.

[stefan-it]

Like this: https://github.com/bigscience-workshop/promptsource :thinking:

[stefan-it]

Ok, here's my favorite one: https://twitter.com/dvilasuero/status/1641164559888142336 (powered by @dvsrepo) :hugs:

[stefan-it]

We use Argilla now: https://github.com/LEL-A/GerAlpacaDataCleaned/pull/6

It also allows us to add metadata (translation model and original id) as well as sentence embeddings. Argilla itself allows us to label/flag a certain example into several categories, which can be seen as more sophisticated as just a review_needed flag.

[axenov]

Have you seen https://github.com/thisserand/alpaca-lora-finetune-language? There the dataset was translated into German via Google Translator, DeepL and GPT 3.5. What do you think about including them in the dataset as different translation versions?