Report the LLM answers

[mircealungu]

Given that an LLM is an evolving system, reporting the prompts might not be enough: different versions are likely to return different answers. For complete transparency a researcher, when possible, should also report the LLM answers. In this sense, the way a LLM has to be treated is similar to the way a human participant should be treated. Both might provide different answers if asked the same questions at different times.

[sbaltes]

Thanks, I fully agree. There's currently a TODO for this in this section: https://llm-guidelines.org/guidelines/#report-prompts-and-their-development If you like, you can change that subsection and I'd be happy to double-check the changes.

[mircealungu]

If we agree with the main thrust of the idea expressed in the issue description, i can take that paragraph and work it in.

[mircealungu]

Maybe another question here: if we insert it in the TODO placeholder we'll have to update the guideline to say report prompts and answers. Alternatively, we could add a separate guideline that would focus specifically onto the report answers idea.

[sbaltes]

I agree with the general sentiment of the issue description. I would add something to:

In this sense, the way a LLM has to be treated is similar to the way a human participant should be treated. Both might provide different answers if asked the same questions at different times.

The difference is that, while for human participants conversations often cannot be reported due to confidentiality, LLM conversations can.

[sbaltes]

Feel free to update the section headings as well!