How to evaluate the quality of generated text

thorinf / simple-diffusion-lm

A simple DIffusion LM approach.

MIT License

20 stars 1 forks source link

How to evaluate the quality of generated text #8

Closed bansky-cl closed 1 year ago

bansky-cl commented 1 year ago

What great and simple code! But I have some confusion on it.

when i train the model, each epoch will generate 'num_examples=8' samples. how can i evaluate the quality of generated text ? Or you can tell me some metrics to evaluate it.

thorinf commented 1 year ago

This is something i need to add. So unconditioned text generation is quite hard to get metrics on, scores like BLEU require reference text which unconditioned doesnt have.

What I need to do is add conditional generation support and ask the model the complete the sentence, then I can use test data to evaluate the model.

bansky-cl commented 1 year ago

thks. I also will try some conditional text diffusion. looking forward to your implementation on conditional gen.

thorinf commented 1 year ago

Just want to say that current results from this aren't spectacular, its been hard to get good sentences from any variants I've worked on. I've also tried other methods/repos, and it's a similar story