how to test bb2 long memory?

[andysun-sw]

I found that BB2's memory is very short and it's difficult to remember more than five sentences of dialogue. It seems that [beyond goldfish memory long term open domain conversation] doesn't work. my question is: 1,how to test bb2 long memory, I use following command and it doesn't work, did i make the right argument? Or should I add more argument in command line: parlai interactive -mf zoo:blenderbot2/blenderbot2_3B/model --search-server relevant_search_server --knowledge-access-method memory_only 2,If I have a conversation that needs to be accurately recorded by the system, where would it be better to add it to the program? At first, I wanted to add to the history dialogue, but I can't find the code place. 3, how to finetune?If I have a conversation that needs to be accurately recorded by the system, do I need to train this by finetune way?

[klshuster]

1. that command should work as posted
2. I'm not sure I understand this question - do you want to provide BB with prior history? Or record your conversation with BB?
3. Again, not sure what you are asking here; you want to train the model on one conversation?

[andysun-sw]

for example, I have chated with one person for one session, and I want the bb2 know that dialogue and continue with that person. I don't want bb2 to repeat the same dialogue with that person. So I want bb2 remember the dialogue first.

[klshuster]

you can simulate BB2 having seen the conversation by pasting it in as your first message after running parlai interactive

[renzs6]

Hi @klshuster what's your recommended approach for having BB2 see previous conversations on other chat services i.e. websocket implementation? would pasting it during run time/inferencing work? or are there any modifications needed

[klshuster]

repeatedly calling agent.observe in the code will have BB2 build up the conversational history; pasting also works as well

[github-actions[bot]]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.

[tungnt55]

repeatedly calling agent.observe in the code will have BB2 build up the conversational history; pasting also works as well

If we paste the dialog history into the first turn of the new conversation, how should we specify which utterance is from the user and which one is from the chatbot? Or they are not differentiated at all?

[klshuster]

they are not differentiated; in fact, the model will work backwards and assume that the most recent message in the conv. history is from the user, then the message before that the bot, etc.

[tungnt55]

Thanks for the answer. I am still not clear how the model handles the conversation history. For example we have a conversation history like this:
User: Hello, my name is Tom.
System: Hello Tom, nice to meet you! Where are you from?
User: I am from the Singapore. I moved to Australia last year.

In the interactive command line mode, if we paste the whole history as the first message of the current conversation, it will look like:
"Hello, my name is Tom. Hello Tom, nice to meet you! Where are you from? I am from the Singapore. I moved to Australia last year."
In that case, how does the model distinguish the messages in each turn of the conversation history?

[klshuster]

you would need to use delimiters to separate context messages. BB2 3B's delimiter is a double space, and BB2 400M's delimiter is a newline.

For the former, you would input (note the double spaces): Hello, my name is Tom. Hello Tom, nice to meet you! Where are you from? I am from the Singapore. I moved to Australia last year.

For the latter, you would input: Hello, my name is Tom.\nHello Tom, nice to meet you! Where are you from?\nI am from the Singapore. I moved to Australia last year.