Possible Extensions

[anupamme]

Hello,

This is not an issue but more like a feature request or asking for advise.

I am looking to use/extend chirpy cardinal for the following use case:

A social bot that can take reviews/feedback for a particular item e.g. if I am running an online cooking class and someone participated in one of them and now I want to take feedback on that class, so what I want precisely bot to do:

1. Pick one of the attributes/features to ask review on e.g. chef's preparedness, availability of ingredients, overall experience, the outcome of the class
2. Have a bit of in-detail conversation on any of the attributes. I want to measure the performance of the bot on how much detailed, nuanced information it can get from the user. So the metric is more qualitative than quantitative.
3. To be able to use as much context in the conversation e.g. if I provide other people's reviews about the same attribute then can the bot use those reviews as well in the conversation e.g. I am sorry and surprised to hear you did not like the attention to detail by the chef, because other people have rated the chef positively on it.
4. So I am looking for the capability where the bot can have a more detailed (or somewhat meaningful) conversation on any of the attribute.

How would you advise me to go build these features in the existing implementation?

[AshwinParanjape]

This sounds like a really cool use case. The main choice point you have is between handwritten templates and neural generation. Handwritten templates work fine when the kinds of bot and user responses are enumerable. When they become open-ended, handwritten templates and regex classifiers break. You have multiple aspects: attributes/features (1), qualitative user feedback (2) and external context (3). Am I correct in saying that the attributes/features are enumerable but nuanced feedback and external context are open-ended? If that's the case, you can try the following

• Use Neural Chat to get nuanced feedback. Currently it is setup to have a 2-3 turn discussion on topics like food, current activities etc. It starts with a question, which in your case will be the particular attribute and then it'll continue the conversation. There are limits to it, because it was trained on empathetic dialogue it might not generalize to the attributes you want. You can look at existing neural chat treelets here https://github.com/stanfordnlp/chirpycardinal/tree/main/chirpy/response_generators/neural_chat/treelets
• To add context to the conversation, you can use the ConvPara callable. It is trained to take background, which would be text like "People have rated positively on " and an entity which is typically the central entity so perhaps maps well to "" (see https://github.com/stanfordnlp/chirpycardinal/blob/635957802720467cd4447caa9481ad6152f10bd3/chirpy/annotators/convpara.py#L51) . It'll generate paraphrases which conversationally try to add the kind of sentence you described. Your milage may vary because it was trained with Reddit's Today I Learned subreddit and the Topical Chat dataset, where most of the facts are surprising. So it might say stuff like "Did you know ..." or "I found out ...". It should be possible to modify the remote callable to take a prefix, and so that might give you some degree of transfer to your domain. In that case you provide a prefix like "I am sorry and surprised to hear", and it might produce the rest.

I made many assumptions, but feel free to reach out further.

[anupamme]

Hello,

The assumptions you made are close to what I am looking for so you are correct in saying that attributes/features are enumerable, they may have sub-attributes (so there is hierarchy) in the sense that when you are talking about the chef preparedness, you want to know attention to detail on ingredients, proper instructions, attention during the class.

1. So the way I am thinking of designing it: have a response generator for each attribute e.g. chef preparedness, usability of the online tool. And have a treelet for each sub-attribute e.g. ingredients, instructions, class experience. Does this design look okay to you?

2. When you said 2-3 turn discussion, is that for response generator or a treelet? Can I setup a 2-3 turn discussion for each treelet?

3. On using ConvPara, I see it is being used used by treelets in wiki RG, however it is not very clear how to use the function get_paraphrases. I mean I would like to just test this function with different arguments and get a sense of what input works best in my case. Is there a way to do design and run these tests? If not, would it be possible to provide some sample inputs and outputs for this function.

Btw, I just read the article at [1] and it helped me a lot in my understanding :).

1. https://ai.stanford.edu/blog/chirpy-cardinal/

[AshwinParanjape]

1. The response generators (RG) are designed to have priority levels to be able to interrupt each other via an API . The RGs are loosely coupled, different people can write different RGs without knowing other RGs' behaviours (to a large extent). Within an RG, you can do whatever you want. Treelets are an easy way of creating a dialogue graph. And so the treelets are more expressive in terms of dialogue flow, but while writing treelets (within an RG) you need to know about other treelets, so they are less modular. I think in your case, you might be better off writing a single RG (because you are in complete control) and writing treelets for attributes and subattributes.

2) Yes. An example here https://github.com/stanfordnlp/chirpycardinal/blob/main/chirpy/response_generators/neural_chat/treelets/food_treelet.py

3) The convpara annotator just a convenience wrapper around the service running in convpara docker container. If you want to play around and see how good the model is, you can directly query the service. It takes the following as a json input https://github.com/stanfordnlp/chirpycardinal/blob/635957802720467cd4447caa9481ad6152f10bd3/chirpy/annotators/convpara.py#L70 with the following default config https://github.com/stanfordnlp/chirpycardinal/blob/635957802720467cd4447caa9481ad6152f10bd3/chirpy/annotators/convpara.py#L15 Let me know if you have any trouble with querying the convpara service.

Since you are looking into a lot of detail, the technical paper might also help in understanding the structure of the bot and the different RGs as well - https://arxiv.org/abs/2008.12348

[anupamme]

"Tree-lets are an easy way of creating a dialogue graph. And so the tree-lets are more expressive in terms of dialogue flow, but while writing tree-lets (within an RG) you need to know about other tree-lets, so they are less modular. "

How do we make a dialogue graph/flow? Does a tree-let correspond to a node (or path) in the graph? Like a node knows its neighbours a tree-let also knows its neighbouring tree-lets, is this the correct understanding?

I was looking at Figure 2 (page 10) of the technical paper to understand the dialogue graph, it illustrates the dialogue flow for the Movies RG. I did not find Movies RG in the codebase, at the same time I could not find the dialogue graph for any other RG, I would like to see it for NeuralChat.

This comment is just to help me understand with lesser effort, however, if answering this question is too much work for you then a high-level explanation would suffice.

[AshwinParanjape]

We had to take out Movies RG because it used one of Amazon's internal APIs. To be honest, there is no consistent definition of treelets, but roughly it corresponds to a node in the graph and handles one turn at a time. It typically takes in a user response, generate a bot response, is able to classify user responses and accordingly hand over control to next treelet (which can include itself as well).

For NeuralChat there are 5-6 treelets each handles, entry into the topic, exit from the topic and multiple neural generated responses in between. I had pointed to one of the neural chat treelets earlier but maybe you missed it in the wall of text. Here's a treelet to talk about foods https://github.com/stanfordnlp/chirpycardinal/blob/main/chirpy/response_generators/neural_chat/treelets/food_treelet.py

It inherits from an abstract neural treelet: https://github.com/stanfordnlp/chirpycardinal/blob/main/chirpy/response_generators/neural_chat/treelets/abstract_treelet.py

For you to write a new treelet, you can use foods as a template and modify the starter phrases.

Does that help understand?

[anupamme]

"It typically takes in a user response, generate a bot response, is able to classify user responses and accordingly hand over control to next treelet (which can include itself as well)."

Where in the code does a treelet hands over the to next treelet (e.g. food_treelet)? I looked at food_treelet and abstract_treelet but could not find it.

I looked at various treelets: food is a good start, emotions_treelet performs the classification based on user utterance and decides accordingly. So writing a treelet is clear to me. Thanks for all the pointers.

[ameliahardy]

Where in the code does a treelet hands over the to next treelet (e.g. food_treelet)? I looked at food_treelet and abstract_treelet but could not find it.

The neural chat RG is a bit different because it selects one treelet at the beginning of its turn and then stays within that treelet until another RG is chosen. This is because each time neural chat is used, it only asks a single scripted question before switching over to giving generative responses.

A couple RGs which use multiple treelets are categories and wiki.

The categories rg decides which treelet to use in this file: https://github.com/stanfordnlp/chirpycardinal/blob/main/chirpy/response_generators/categories/categories_response_generator.py

And the wiki rg has an example of how treelets can be selected here: https://github.com/stanfordnlp/chirpycardinal/blob/main/chirpy/response_generators/wiki/wiki_response_generator.py

If you'd like to set the next treelet within the current treelet, rather than within the main logic for the RG, one way to do that is by having the current treelet update the RG's conditional state by setting state.cur_treelet = treelet you want to go to next

[anupamme]

Closing this as I have no further questions right now.