Build benchmark to evaluate text->image generative models

[rom1504]

I think it would be really interesting to collect a list of prompts for each "text->image" task, eg

• illustrations
• realistic
• compositions
• many classes
• faces/people etc

and then turn that into a benchmark so we would know what each model strengths really are

[rom1504]

I'm sure I'm not the first one to have this idea, so the first task would be collecting papers that did things on the topic

[rom1504]

similar to https://github.com/EleutherAI/lm-evaluation-harness

[christophschuhmann]

here’s the plan https://docs.google.com/document/d/18uAhB_99lTsfhKkBd6tiKbMhZcQ9hErLgJqEWuTOcQY/edit

[rom1504]

That seems to be a plan for evaluating the retrieval task for https://github.com/LAION-AI/project-menu/issues/13

Here is for generative models

[mehdidc]

This would be super cool. I haven't read it yet but this paper is very relevant to this https://arxiv.org/abs/2202.04053