BlenderBot2 - search_only assertion error

rerngvit commented 2 years ago

Bug description It seems that the agent BlenderBot2 with the option knowledge-access-method set to search_only will have an assertion error at run time. (I have tested for the default access method classify, which works normally as expected.)

Reproduction steps

Install ParlAI for development locally
1. Clone the repository with the main branch, e.g., git clone https://github.com/facebookresearch/ParlAI
2. Create a virtual environment locally, e.g., python3 -m venv dev_parlai
3. Activate the virtual environment, e.g., source dev_parlai/bin/activate
4. Install package dependency for ParlAI, e.g., pip install -r requirement.txt
5. Setup ParlAI for development, e.g., python setup.py develop
Start a ParlAI search server in a separate bash shell, e.g., this one
1. Clone the repository, e.g., git clone https://github.com/JulesGM/ParlAI_SearchEngine
2. Create another virtual environment locally, e.g., python3 -m venv dev_search_server
3. Activate the virtual environment, e.g., source dev_search_server/bin/activate
4. Install package dependency for the search server, e.g, pip install -r requirement.txt
5. Start the search server: python search_server.py serve --host 0.0.0.0:8080
Test the ParlAI installation
1. parlai interactive --model-file zoo:blenderbot2/blenderbot2_400M/model --search_server 0.0.0.0:8080 --knowledge-access-method search_only
2. You will reach the chat interactive prompt Enter Your Message:
3. Just type any message, e.g., hello

Expected behavior The bot should respond with normal conversation like when knowledge-access-method is not specified.

Logs Please paste the command line output:

Enter Your Message: hello
Traceback (most recent call last):
  File "/Users/ereryan/venvs/dev_parlAI_2021/bin/parlai", line 33, in <module>
    sys.exit(load_entry_point('parlai', 'console_scripts', 'parlai')())
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/__main__.py", line 14, in main
    superscript_main()
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/core/script.py", line 325, in superscript_main
    return SCRIPT_REGISTRY[cmd].klass._run_from_parser_and_opt(opt, parser)
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/core/script.py", line 108, in _run_from_parser_and_opt
    return script.run()
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/scripts/interactive.py", line 118, in run
    return interactive(self.opt)
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/scripts/interactive.py", line 93, in interactive
    world.parley()
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/tasks/interactive/worlds.py", line 89, in parley
    acts[1] = agents[1].act()
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/core/torch_agent.py", line 2143, in act
    response = self.batch_act([self.observation])[0]
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/core/torch_agent.py", line 2239, in batch_act
    output = self.eval_step(batch)
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/projects/blenderbot2/agents/blenderbot2.py", line 782, in eval_step
    output = super().eval_step(batch)
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/agents/rag/rag.py", line 290, in eval_step
    output = super().eval_step(batch)
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/core/torch_generator_agent.py", line 876, in eval_step
    batch, self.beam_size, maxlen, prefix_tokens=prefix_tokens
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/agents/rag/rag.py", line 673, in _generate
    gen_outs = self._rag_generate(batch, beam_size, max_ts, prefix_tokens)
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/agents/rag/rag.py", line 713, in _rag_generate
    self, batch, beam_size, max_ts, prefix_tokens
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/parlai/core/torch_generator_agent.py", line 1094, in _generate
    encoder_states = model.encoder(*self._encoder_input(batch))
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/projects/blenderbot2/agents/modules.py", line 821, in encoder
    segments,
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/projects/blenderbot2/agents/modules.py", line 226, in encoder
    num_memory_decoder_vecs,
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/projects/blenderbot2/agents/modules.py", line 357, in retrieve_and_concat
    search_queries, query_vec, search_indices
  File "/Users/ereryan/Dropbox/coding_workspace/analytic-workspace/chatbots/parlai-github/projects/blenderbot2/agents/modules.py", line 513, in perform_search
    assert search_queries
AssertionError

Additional context Add any other context about the problem here. (like proxy settings, network setup, overall goals, etc.)

klshuster commented 2 years ago

Hi there - if you would like to use --knowledge-access-method search_only, you'll need to provide a search query generator that always generates a search query; the default for BlenderBot2 can additionally generate a token indicating memory retrieval.

I would specify --query-generator-model-file zoo:sea/bart_sq_gen/model, as this is the search query generator used in the internet-augmented dialogue generation project

rerngvit commented 2 years ago

Thank you. Specifying the query generator model did resolve the error.

facebookresearch / ParlAI

BlenderBot2 - search_only assertion error #4016