[Project Idea] InstructLab Taxonomy Reporting

[hemajv]

Currently, InstructLab does not publish any metrics per taxonomy leaf node. We would like to explore different ways we can evaluate the InstructLab model being fine tuned via the taxonomy approach and come up with metrics to evaluate each of the taxonomy leaf nodes.

Each leaf node in the taxonomy represents one particular skill, or set of knowledge. Here is one example: https://github.com/instructlab/taxonomy/blob/main/compositional_skills/linguistics/complete_common_expressions/qna.yaml. We see that each leaf has question and answer pairs. We would like to track how many questions from these yaml files the model answers correctly (for some definition of correctness), over time.

[hemajv]

@erikerlandson please feel free to add to this

[PalmPalm7]

First steps according to Sanjay, feel free to edit/add!

1. Run Granite model
2. Take Granite model to qLora fine-tune on small datasets
3. Testing infra and learn a few things
4. Then evaluate knowledge trees