search

facebookresearch / ParlAI

A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
https://parl.ai
MIT License
10.47k stars 2.09k forks source link

BlenderBot2 allowed options #3877

Closed sjscotti closed 3 years ago

sjscotti commented 3 years ago

Hi! I have started evaluating BlenderBot2 and would like to try different combinations of parlai options that are allowed for this model. For example, since I am using an external —search-server that shows me when the bot does internet searches, I have noticed the first question I put to the bot does not trigger a query. So I am assuming the default is to switch between the internal corpus, and external searches. I would like to evaluate the bot with the bot always doing searches. So knowing how to set such a flag would be helpful.

UPDATE, I tried the flag --knowledge_access_method all and --knowledge_access_method server_only for the example I quoted above and, for using either of these, I got the following error...

Enter [DONE] if you want to end the episode, [EXIT] to quit.
09:28:20 | creating task(s): interactive
Enter Your Message: Who is the president?
Traceback (most recent call last):
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Steve\anaconda3\envs\ParlAI\Scripts\parlai.exe\__main__.py", line 7, in <module>
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\__main__.py", line 14, in main
    superscript_main()
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\core\script.py", line 325, in superscript_main
    return SCRIPT_REGISTRY[cmd].klass._run_from_parser_and_opt(opt, parser)
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\core\script.py", line 108, in _run_from_parser_and_opt
    return script.run()
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\scripts\interactive.py", line 118, in run
    return interactive(self.opt)
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\scripts\interactive.py", line 93, in interactive
    world.parley()
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\tasks\interactive\worlds.py", line 89, in parley
    acts[1] = agents[1].act()
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\core\torch_agent.py", line 2143, in act
    response = self.batch_act([self.observation])[0]
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\core\torch_agent.py", line 2239, in batch_act
    output = self.eval_step(batch)
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\projects\blenderbot2\agents\blenderbot2.py", line 782, in eval_step
    output = super().eval_step(batch)
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\core\torch_generator_agent.py", line 875, in eval_step
    beam_preds_scores, beams = self._generate(
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\agents\rag\rag.py", line 657, in _generate
    gen_outs = self._rag_generate(batch, beam_size, max_ts, prefix_tokens)
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\agents\rag\rag.py", line 696, in _rag_generate
    return self._generation_agent._generate(
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\parlai\core\torch_generator_agent.py", line 1094, in _generate
    encoder_states = model.encoder(*self._encoder_input(batch))
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\projects\blenderbot2\agents\modules.py", line 802, in encoder
    enc_out, mask, input_turns_cnt, top_docs, top_doc_scores = super().encoder(  # type: ignore
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\projects\blenderbot2\agents\modules.py", line 211, in encoder
    expanded_input, top_docs, top_doc_scores = self.retrieve_and_concat(
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\projects\blenderbot2\agents\modules.py", line 353, in retrieve_and_concat
    search_docs, search_doc_scores = self.perform_search(
  File "C:\Users\Steve\anaconda3\envs\ParlAI\lib\site-packages\projects\blenderbot2\agents\modules.py", line 509, in perform_search
    assert search_queries
AssertionError

I didn't get any error when not using this flag. Thanks in advance!

klshuster commented 3 years ago

setting --knowledge-access-method search_only is the correct way to go about this; would you mind sharing your full command?

another way to make sure that the model always generates a search query is to specify a --query-generator-model-file for a query generator trained only to generate search queries (the BB2 one is trained to predict whether to access long-term memory as well). Try setting --query-generator-model-file zoo:sea/bart_sq_gen/model

sjscotti commented 3 years ago

Thanks for getting back with me! Here is the command I used and the subsequent output from the bot -- up until the output I showed in my original post above. 


(ParlAI) C:\Users\Steve>parlai interactive -mf zoo:blenderbot2/blenderbot2_3B/model --search-server 127.0.0.1:8080  --knowledge-access-method search_only
09:27:18 | Overriding opt["search_server"] to 127.0.0.1:8080 (previously: None)
09:27:18 | Overriding opt["knowledge_access_method"] to search_only (previously: classify)
09:27:18 | Using CUDA
09:27:18 | loading dictionary from C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/blenderbot2_3B/model.dict
09:27:18 | num words = 8008
09:27:18 | BlenderBot2Fid: full interactive mode on.
09:27:44 | Creating the search engine retriever.
09:27:44 | No portocol provided, using "http://"
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
09:27:51 | Building Query Generator from file: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/query_generator/model
09:27:59 | Building Memory Decoder from file: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/memory_decoder/model
09:28:14 | Total parameters: 3,022,943,744 (3,021,108,736 trainable)
09:28:14 | Loading existing model params from C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/blenderbot2_3B/model
09:28:19 | Opt:
09:28:19 |     activation: relu
09:28:19 |     adafactor_eps: '[1e-30, 0.001]'
09:28:19 |     adam_eps: 1e-08
09:28:19 |     add_p1_after_newln: False
09:28:19 |     allow_missing_init_opts: False
09:28:19 |     attention_dropout: 0.0
09:28:19 |     batchsize: 16
09:28:19 |     beam_block_full_context: False
09:28:19 |     beam_block_list_filename: None
09:28:19 |     beam_block_ngram: 3
09:28:19 |     beam_context_block_ngram: 3
09:28:19 |     beam_delay: 30
09:28:19 |     beam_length_penalty: 0.65
09:28:19 |     beam_min_length: 20
09:28:19 |     beam_size: 10
09:28:19 |     betas: '[0.9, 0.999]'
09:28:19 |     bpe_add_prefix_space: None
09:28:19 |     bpe_debug: False
09:28:19 |     bpe_dropout: None
09:28:19 |     bpe_merge: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/blenderbot2_3B/model.dict-merges.txt
09:28:19 |     bpe_vocab: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/blenderbot2_3B/model.dict-vocab.json
09:28:19 |     candidates: inline
09:28:19 |     cap_num_predictions: 100
09:28:19 |     codes_attention_num_heads: 4
09:28:19 |     codes_attention_type: basic
09:28:19 |     compressed_indexer_factory: IVF4096_HNSW128,PQ128
09:28:19 |     compressed_indexer_gpu_train: False
09:28:19 |     compressed_indexer_nprobe: 64
09:28:19 |     compute_tokenized_bleu: False
09:28:19 |     converting: False
09:28:19 |     data_parallel: False
09:28:19 |     datapath: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data
09:28:19 |     datatype: train:stream
09:28:19 |     delimiter: '  '
09:28:19 |     dict_class: parlai.core.dict:DictionaryAgent
09:28:19 |     dict_endtoken: __end__
09:28:19 |     dict_file: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/blenderbot2_3B/model.dict
09:28:19 |     dict_initpath: None
09:28:19 |     dict_language: english
09:28:19 |     dict_loaded: True
09:28:19 |     dict_lower: False
09:28:19 |     dict_max_ngram_size: -1
09:28:19 |     dict_maxtokens: -1
09:28:19 |     dict_minfreq: 0
09:28:19 |     dict_nulltoken: __null__
09:28:19 |     dict_starttoken: __start__
09:28:19 |     dict_textfields: text,labels
09:28:19 |     dict_tokenizer: bytelevelbpe
09:28:19 |     dict_unktoken: __unk__
09:28:19 |     display_add_fields:
09:28:19 |     display_examples: False
09:28:19 |     display_prettify: False
09:28:19 |     doc_chunk_split_mode: word
09:28:19 |     doc_chunks_ranker: head
09:28:19 |     download_path: None
09:28:19 |     dpr_model_file: zoo:hallucination/bart_rag_token/model
09:28:19 |     dpr_num_docs: 25
09:28:19 |     dropout: 0.0
09:28:19 |     dynamic_batching: None
09:28:19 |     embedding_projection: random
09:28:19 |     embedding_size: 2560
09:28:19 |     embedding_type: random
09:28:19 |     embeddings_scale: True
09:28:19 |     encode_candidate_vecs: True
09:28:19 |     encode_candidate_vecs_batchsize: 256
09:28:19 |     eval_candidates: inline
09:28:19 |     ffn_size: 10240
09:28:19 |     fixed_candidate_vecs: reuse
09:28:19 |     fixed_candidates_path: None
09:28:19 |     force_fp16_tokens: True
09:28:19 |     fp16: True
09:28:19 |     fp16_impl: mem_efficient
09:28:19 |     generation_model: transformer/generator
09:28:19 |     gold_document_key: __selected-docs__
09:28:19 |     gold_document_titles_key: select-docs-titles
09:28:19 |     gold_knowledge_passage_key: checked_sentence
09:28:19 |     gold_knowledge_title_key: title
09:28:19 |     gold_sentence_key: __selected-sentences__
09:28:19 |     gpu: -1
09:28:19 |     gradient_clip: 0.1
09:28:19 |     hide_labels: False
09:28:19 |     history_add_global_end_token: end
09:28:19 |     history_reversed: False
09:28:19 |     history_size: -1
09:28:19 |     hnsw_ef_construction: 200
09:28:19 |     hnsw_ef_search: 128
09:28:19 |     hnsw_indexer_store_n: 128
09:28:19 |     ignore_bad_candidates: False
09:28:19 |     image_cropsize: 224
09:28:19 |     image_mode: raw
09:28:19 |     image_size: 256
09:28:19 |     indexer_buffer_size: 65536
09:28:19 |     indexer_type: compressed
09:28:19 |     inference: beam
09:28:19 |     init_fairseq_model: None
09:28:19 |     init_model: /private/home/kshuster/ParlAI/data/models/blender/blender_3B/model
09:28:19 |     init_opt: None
09:28:19 |     insert_gold_docs: True
09:28:19 |     interactive_candidates: fixed
09:28:19 |     interactive_mode: True
09:28:19 |     interactive_task: True
09:28:19 |     invsqrt_lr_decay_gamma: -1
09:28:19 |     is_debug: False
09:28:19 |     knowledge_access_method: search_only
09:28:19 |     label_truncate: 128
09:28:19 |     learn_embeddings: True
09:28:19 |     learn_positional_embeddings: False
09:28:19 |     learningrate: 1e-05
09:28:19 |     local_human_candidates_file: None
09:28:19 |     log_keep_fields: all
09:28:19 |     loglevel: info
09:28:19 |     lr_scheduler: reduceonplateau
09:28:19 |     lr_scheduler_decay: 0.5
09:28:19 |     lr_scheduler_patience: 3
09:28:19 |     max_doc_token_length: 64
09:28:19 |     memory_attention: sqrt
09:28:19 |     memory_decoder_beam_min_length: 10
09:28:19 |     memory_decoder_beam_size: 3
09:28:19 |     memory_decoder_delimiter: '\n'
09:28:19 |     memory_decoder_ignore_phrase: persona:
09:28:19 |     memory_decoder_key: full_text
09:28:19 |     memory_decoder_model_file: zoo:blenderbot2/memory_decoder/model
09:28:19 |     memory_decoder_one_line_memories: False
09:28:19 |     memory_decoder_truncate: -1
09:28:19 |     memory_doc_delimiter: :
09:28:19 |     memory_doc_title_delimiter: ' / '
09:28:19 |     memory_extractor_phrase: persona:
09:28:19 |     memory_key: personas
09:28:19 |     memory_reader_model: None
09:28:19 |     memory_retriever_truncate: -1
09:28:19 |     memory_writer_model: bert
09:28:19 |     memory_writer_model_file: zoo:hallucination/multiset_dpr/hf_bert_base.cp
09:28:19 |     min_doc_token_length: 64
09:28:19 |     model: projects.blenderbot2.agents.blenderbot2:BlenderBot2FidAgent
09:28:19 |     model_file: C:\Users\Steve\anaconda3\envs\ParlAI\Lib\site-packages\data\models\blenderbot2/blenderbot2_3B/model
09:28:19 |     model_parallel: True
09:28:19 |     momentum: 0
09:28:19 |     multitask_weights: '[3.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]'
09:28:19 |     n_decoder_layers: 24
09:28:19 |     n_docs: 5
09:28:19 |     n_encoder_layers: 2
09:28:19 |     n_extra_positions: 0
09:28:19 |     n_heads: 32
09:28:19 |     n_layers: 2
09:28:19 |     n_positions: 128
09:28:19 |     n_ranked_doc_chunks: 1
09:28:19 |     n_segments: 0
09:28:19 |     nesterov: True
09:28:19 |     no_cuda: False
09:28:19 |     normalize_sent_emb: False
09:28:19 |     nus: [0.7]
09:28:19 |     optimizer: mem_eff_adam
09:28:19 |     outfile:
09:28:19 |     output_conversion_path: None
09:28:19 |     output_scaling: 1.0
09:28:19 |     override: "{'search_server': '127.0.0.1:8080', 'knowledge_access_method': 'search_only'}"
09:28:19 |     parlai_home: /private/home/kshuster/ParlAI
09:28:19 |     path_to_dense_embeddings: None
09:28:19 |     path_to_dpr_passages: zoo:hallucination/wiki_passages/psgs_w100.tsv
09:28:19 |     path_to_index: zoo:hallucination/wiki_index_compressed/compressed_pq
09:28:19 |     person_tokens: False
09:28:19 |     poly_attention_num_heads: 4
09:28:19 |     poly_attention_type: basic
09:28:19 |     poly_faiss_model_file: None
09:28:19 |     poly_n_codes: 64
09:28:19 |     poly_score_initial_lambda: 0.5
09:28:19 |     polyencoder_init_model: wikito
09:28:19 |     polyencoder_type: codes
09:28:19 |     print_docs: False
09:28:19 |     query_generator_beam_min_length: 2
09:28:19 |     query_generator_beam_size: 1
09:28:19 |     query_generator_delimiter: '\n'
09:28:19 |     query_generator_ignore_phrase: persona:
09:28:19 |     query_generator_inference: beam
09:28:19 |     query_generator_key: full_text
09:28:19 |     query_generator_model_file: zoo:blenderbot2/query_generator/model
09:28:19 |     query_generator_truncate: -1
09:28:19 |     query_model: bert_from_parlai_rag
09:28:19 |     rag_model_type: token
09:28:19 |     rag_query_truncate: 512
09:28:19 |     rag_retriever_query: full_history
09:28:19 |     rag_retriever_type: search_engine
09:28:19 |     rag_turn_discount_factor: 1.0
09:28:19 |     rag_turn_marginalize: doc_then_turn
09:28:19 |     rag_turn_n_turns: 2
09:28:19 |     rank_candidates: False
09:28:19 |     rank_top_k: -1
09:28:19 |     reduction_type: mean
09:28:19 |     regret: False
09:28:19 |     regret_intermediate_maxlen: 32
09:28:19 |     regret_model_file: None
09:28:20 |     relu_dropout: 0.0
09:28:20 |     repeat_blocking_heuristic: True
09:28:20 |     retriever_debug_index: None
09:28:20 |     retriever_delimiter: '\n'
09:28:20 |     retriever_embedding_size: 768
09:28:20 |     retriever_ignore_phrase: persona:
09:28:20 |     return_cand_scores: False
09:28:20 |     save_format: conversations
09:28:20 |     search_query_generator_beam_min_length: 2
09:28:20 |     search_query_generator_beam_size: 1
09:28:20 |     search_query_generator_inference: greedy
09:28:20 |     search_query_generator_model_file: zoo:blenderbot2/query_generator/model
09:28:20 |     search_query_generator_text_truncate: 512
09:28:20 |     search_server: 127.0.0.1:8080
09:28:20 |     share_encoders: True
09:28:20 |     share_search_and_memory_query_encoder: False
09:28:20 |     share_word_embeddings: True
09:28:20 |     single_turn: False
09:28:20 |     skip_generation: False
09:28:20 |     skip_retrieval_token: no_passages_used
09:28:20 |     special_tok_lst: None
09:28:20 |     split_lines: False
09:28:20 |     splitted_chunk_length: 256
09:28:20 |     starttime: Jul09_14-08
09:28:20 |     t5_dropout: 0.0
09:28:20 |     t5_generation_config: None
09:28:20 |     t5_model_arch: t5-base
09:28:20 |     t5_model_parallel: False
09:28:20 |     task: None
09:28:20 |     temperature: 1.0
09:28:20 |     text_truncate: 128
09:28:20 |     tfidf_max_doc_paragraphs: -1
09:28:20 |     tfidf_model_path: zoo:wikipedia_full/tfidf_retriever/model
09:28:20 |     thorough: False
09:28:20 |     topk: 10
09:28:20 |     topp: 0.9
09:28:20 |     train_predict: False
09:28:20 |     truncate: 128
09:28:20 |     update_freq: 1
09:28:20 |     use_memories: False
09:28:20 |     use_reply: label
09:28:20 |     variant: prelayernorm
09:28:20 |     verbose: False
09:28:20 |     warmup_rate: 0.0001
09:28:20 |     warmup_updates: 100
09:28:20 |     weight_decay: None
09:28:20 |     wrap_memory_encoder: False
Enter [DONE] if you want to end the episode, [EXIT] to quit.

also, I don't quite understand the difference between the flag --knowledge_access_method all and --knowledge_access_method search_only, but both are giving me the same error.
For your suggestion to use the setting --query-generator-model-file zoo:sea/bart_sq_gen/model , do I not specify anything for the --knowledge_access_method flag?

Thanks!

klshuster commented 3 years ago
  1. --knowledge-access-method all means BB2 will both search and retrieve from memory; --knowledge-access-method search_only will always perform a search. However, the default --query-generator-model-file is one that is designed to predict whether to search or retrieve from memory; what I think is happening here is that the model is being asked to search but the query generator is generating the "retrieve from memory token". This, too, can be remedied by swapping in the zoo:sea/bart_sq_gen/model (or any other query generator you'd like to use)
  2. If specifying --query-generator-model-file zoo:sea/bart_sq_gen/model, you don't need to change the --knowledge-access-method, as the default is classify and that specific query generator will always generate a search query
sjscotti commented 3 years ago

Thanks! I tried both adding --query-generator-model-file zoo:sea/bart_sq_gen/model and --knowledge_access_method search_only together, and just using --query-generator-model-file zoo:sea/bart_sq_gen/model. Both cases did the internet search for each query as you said, but the former seemed to give better answers for the simple questions I was testing it with. It also gave me better answers with -mf zoo:blenderbot2/blenderbot2_400M/model than using the larger -mf zoo:blenderbot2/blenderbot2_3B/model model. However, for my use case - using the bot to answer questions on a specific area of knowledge - I think that just having the internet search for all queries is not going to be sufficient, and I will need to do some additional training (transfer learning?) on the blenderbot2/blenderbot2_400M/model corpus itself.
Thanks for your help!

klshuster commented 3 years ago

Great, will close this issue then - feel free to reopen or file a new one if there are any additional questions