Open dvsrepo opened 9 months ago
Hi!
Sorry for the late response and thanks for pointing that out! Yes, it seems to be a bug, and the ">" should be ">=".
Intuitively, a true 10
score should correspond to high fine-grained scores while a mistaken 10
relates to low ones. We will check all the 2k samples immediately.
Here's a space we've been using to verify this:
(login with: owner/12345678)
The only issue is that there's some examples with overall_score 10 that are good (the majority are bad though).
We've been working on curating this data programmatically and with Argilla and we'd be super happy to contribute back
Thanks for building an amazing project!
Can't wait to see the updated dataset!
Just updated the dataset, please check!
Oh no, on our official page: https://huggingface.co/datasets/openbmb/UltraFeedback
Hi!
Congrats on this amazing project.
We've been exploring the data and identified an issue with very high
overall_score
responses. The issue seems to be related with this line. This causes responses with a critique rating of1
to become a10
. We noticed this by looking at the critique rational which was highly negative for many (~2K) examples with anoverall_score
of10
.