Internet-Augmented Dialogue Generation results with google search yields poor results...

[sjscotti]

Hi I am trying out Internet-Augmented Dialogue Generation (https://github.com/facebookresearch/ParlAI/tree/main/projects/sea) using the some of the examples from the paper at https://arxiv.org/pdf/2107.07566.pdf and the Google search engine of https://github.com/JulesGM/ParlAI_SearchEngine. The search terms sent to Google seem to follow what is in the paper fairly well, but the responses from the dialogue are not nearly as good as what the paper shows. This is the command I am using to run an interactive session...

parlai interactive --model-file zoo:sea/bart_fid_sqse/model --task wizard_of_internet --search-server 127.0.0.1:9080  --search-query-generator-model-file zoo:sea/bart_sq_gen/model     --rag-retriever-type search_engine

Here is an example of one (the "James Bond cocktail") exchange I am getting...

Enter Your Message: What's your favorite cocktail?
[SearchQuerySearchEngineFiD]: I like the martini. What is your favorite cocktail?
Enter Your Message: Oh, I think Martinis are James Bonds drink, right?  How is that made?
[SearchQuerySearchEngineFiD]: I think it's a cocktail made with gin and vodka.
Enter Your Message: Sounds nice, is that a particular kind of martini?
[SearchQuerySearchEngineFiD]: I think it's a cocktail made with gin and vodka.
Enter Your Message: I think that is in the movie Casino Royale.
[SearchQuerySearchEngineFiD]: I think that is in the movie Casino Royale.
Enter Your Message: Yes, who is your favorite Bond and why?
[SearchQuerySearchEngineFiD]: I think that is in the movie Casino Royale.

The search queries generated were : cocktail, james bond martini, martini, casino royale, and james bond, which are fairly close to the ones indicated in figure 3 of the paper. It seems to be repeating my inputs and not paying much attention to the search results it is getting from Google. Results from other examples in the paper were similar in that the bot did not seem to pay too much attention to the internet search results. Am I doing something wrong in my interactive run?

[klshuster]

Perhaps try with -o gen/blenderbot; this will give provide more reasonable decoding parameters

[sjscotti]

Thanks @klshuster for the quick response! I tried this command...

parlai interactive --model-file zoo:sea/bart_fid_sqse/model --task wizard_of_internet --search-server 127.0.0.1:9080  --search-query-generator-model-file zoo:sea/bart_sq_gen/model     --rag-retriever-type search_engine  -o gen/blenderbot

But I got an error saying that I didn't have an opt preset by that name available, so I used the definition of that preset as shown at https://parl.ai/docs/opt_presets.html and then got these results...

Enter Your Message: What's your favorite cocktail?
[SearchQuerySearchEngineFiD]: Cocktails are a great way to relax and have a good time. Do you have a favorite?
Enter Your Message: Oh, I think Martinis are James Bonds drink, right?  How is that made?
[SearchQuerySearchEngineFiD]: I'm not sure how it's made, but I'm sure it's similar to a martini.
Enter Your Message: Sounds nice, is that a particular kind of martini?
[SearchQuerySearchEngineFiD]: I think it's called the Vesper Martini.  What do you like to drink with it?
Enter Your Message: I think that is in the movie Casino Royale.
[SearchQuerySearchEngineFiD]: I've never heard of that one.  Is that the one with Daniel Craig and Eva Green?
Enter Your Message: Yes, who is your favorite Bond and why?
[SearchQuerySearchEngineFiD]: I really like Daniel Craig.  I think he's the best.  Do you like any other Bond movies?

and it used these queries: cocktail, james bonds cocktail, james bond cocktail, casino royale, and james bond.

These are much better results. Is there a reason that the results using the default parameters in sea did not do as well? Should these options be used with any bot? Actually, the reason I gave sea a try was that I was having a similar issue with a blenderbot2 which I fine-tuned that also seemed to ignore the search results, so I wanted to eliminate variables I may have introduced by trying to reproduce dialogues in sea.

[klshuster]

I believe we did not set the generation parameters within the model files themselves for sea, so the gen/blenderbot parameters are more in line with what was used in that paper. The default is greedy decoding which is generally worse than any beam search variant

[github-actions[bot]]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.