instructlab / instructlab-bot

GitHub bot to assist with the taxonomy contribution workflow
Apache License 2.0
14 stars 16 forks source link

Improve precheck output for better review #220

Closed vishnoianil closed 4 months ago

vishnoianil commented 4 months ago

Currently user provided skill yaml has question and answers. Once we run precheck against it, it generates the questions and answer from the model endpoint. Reviewer now needs to go forth between two different files to see the difference between the answer user expected and the answer model generated. It can be bit annoying.

To make the review process bit better, we can add the question, user expected answer and actual answer from model in the same file. That won't require switching between multiple files for review and hopefully makes reviewers like bit less annoying.