Tool for adding new synthetic columns

[farzadab]

This tools allows you to add new columns, by using a template and feeding it into gpt-4o to create a synthetic "ground-truth" response.

$ just enrich_ds -d fixie-ai/boolq-audio -s train -b with_explanation -c explanation -t $HF_WRITE_TOKEN

Loading dataset "fixie-ai/boolq-audio", new column name: "explanation", template:
Passage: {passage}

Question: {question}

Answer: {answer}

Provide a short explanation to the question given the passage that entails the answer.
Processing split "train"...

Map (num_proc=16): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9427/9427 [16:47<00:00,  9.35 examples/s]

[farzadab]

I agree that it's best to integrate these two into the same file. I'll take a look to see if there's a way to combine them in a simplified manner.

[farzadab]

PTAL