[Maintenance] Example of LLM-based user simulator

[isekulic]

Describe the feature Thank you for your work and for adding LLM-based models to the platform. I would be very grateful to see a working example of LLM-based user simulator in the examples.

Expected behavior The expected behavior is a working script that showcases example of evaluating LLM-based user simulator within the framework. For example, like the issue #152 where the script evaluates TUS within the framework, but with newly-added LLM-based models.

Additional context It seems like examples and documentation (READMEs) were not maintained according to the most recent changes (i.e., adding LLM-based models). It is not clear how to use e.g., PipelineAgent and LLM-based models. One example is that LLM_US.init_session(self, goal, example_dialog:str=None) requires goal to be set, but that is not possible from the PipelineAgent class. Thank you for your aid :)

[zqwerty]

Thanks! I will look into this problem this week.

[isekulic]

@zqwerty, great thanks! I'm looking forward to your answer :)

[zqwerty]

I am sorry for the missing of the LLM documentation. I will add a README.

LLM_US is a little bit different from previous user simulators: it is end-to-end and takes the user goal in natural language. So it is not PipelineAgent (that's why I make LLM_US inherit Agent class). You can refer to the unit test function test_LLM_US_RG in llm/user_similator.py for example usage. You can replace the LLaMa model with ChatGPT to get much better performance.

For interaction between LLM_US and other pipeline agents, I will try to write an example script like #152.

[isekulic]

@zqwerty thank you for your comment! Indeed, I've seen the example usage in llm/user_similator.py.

I'm looking forward to see an example script then, which would enable the full interaction (like #152 that would enable evaluation of the LLM simulators and make it comparable to e.g., Table 8 in your paper).

[Ahmed-Mahmod-Salem]

same issue here, could you please give an example of training any agent with LLM usersim?