Feature Request: Support system-style questions (transferred from Taxonomy)

[roy651]

(Issue was moved from here as it touches multiple areas) When defining a new skill in the pattern of question-context-answer it seems that ilab will generate all of those when it creates the augmented dataset. The implication is that they are all dynamically generated.

I would like to propose that ilab will support an additional entry in the skill's yaml allowing for a "system"-style which will preface the question when generating the answer. This part should be static and need not be generated for each new entry. Most likely, it should be static for the entire skill, hence it should reside in the top level of the yaml with the "task_description".

I think that this capability is something that may be required by additional skills as it follows, the now so common, system/user pattern and I assume others would require that too.

[roy651]

@obuzek per your comment on the other thread: I understand the mechanism you describe and I acknowledge that you're utilizing the "system" prompt as part of the SDG. However, I think that, this design choice is something that should be relatively easy to resolve with a suitable instruction to the SDG ("Do not generate this part"...). Assuming technically it's feasible and the discussion is about the need - One might argue that the set of Q&A (and context) samples should suffice for the model to learn the necessary "rules" and pattern of the specific skill. I do think that this need is valid and quite widespread. As the use of LLMs expands and it not necessarily plain conversation-like, and as the complexity of input and output patterns rises, we see frequent deviations of the model from the required patterns. I expect to see others come up with a similar request.

[bjhargrave]

yaml with the "task_description".

The task_description is already placed in the prompt. So I think it can do what you are requesting. It can include instructions to the teacher model.

[roy651]

@bjhargrave This is not well documented but assuming that the prompt is the one that shows up on diagram 3 in the paper, then I'm not sure it's relevant, cause the task description is part of the first line and not directly related to the generated question. Furthermore, I imagine that that both the model that's generating the responses and the evaluator model should get this static text as part of their prompt in a relevant slot.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.