ModuleNotFoundError: No module named 'parlai_internal'

[happysoymilk]

Bug description Hi, I am trying to run the Wikipedia full dialogue model based on my own classed agent (get input from command line and give answers), and I got this error: https://parl.ai/docs/zoo.html#wizard-of-wikipedia-full-dialogue-retrieval-model

Traceback (most recent call last):
  File "agenttest.py", line 36, in <module>
    main()
  File "agenttest.py", line 16, in main
    default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/model')
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 348, in create_agent_from_model_file
    return create_agent_from_opt_file(opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 384, in create_agent_from_opt_file
    model_class = load_agent_module(opt_from_file['model'])
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\loader.py", line 188, in load_agent_module
    my_module = importlib.import_module(module_name)
  File "D:\py-anaconda\envs\parlai_gpu\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'parlai_internal'

[mojtaba-komeili]

Hmm, parlai_internal shouldn't be there. We need to fix that. Thanks for reporting.

[mojtaba-komeili]

Could you also paste the content of your agenttest.py file please?

[mojtaba-komeili]

Looking back at this, could you send me the command that you ran here too? Also, are you trying to have your agent in parlai_internal as explained here?

[happysoymilk]

Hi, the code for agenttest.py is

# %%

from parlai.core.params import ParlaiParser
from parlai.core.agents import create_agent_from_model_file
from parlai.agents.local_human.local_human import LocalHumanAgent
from agent import RRagent

# %%
import torch
torch.cuda.empty_cache()

def main():

    default1 = create_agent_from_model_file(model_file='zoo:tutorial_transformer_generator/model',opt_overrides={'optimizer':'adam'})
    default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/mode')
    default3 = create_agent_from_model_file(model_file='zoo:hallucination/bart_fid_dpr/model')
    #opt_overrides={'doc_chunk_split_mode': ' word ', "eval_candidates": "vocab"} for poly

    agent1 = RRagent(default1)
    agent2 = RRagent(default2)
    agent3 = RRagent(default3)

    while (not agent1.finished):
        question = input('Play with me, enter [DONE] for ending the episode, enter [EXIT] for ending the convo: \n>')
        try:
            print("agent1:{}".format(agent1.answer(question)))
            print("agent2:{}".format(agent2.answer(question)))
            print("agent3:{}".format(agent3.answer(question)))
        except StopIteration:
            if not agent1.finished:
                print("let's play again")
            else:
                print("Bye bye!")
if __name__ == '__main__':
    main()

the RRagent is a class I defined for my own agent.

[klshuster]

We provide a special script for interacting with the full dialogue retrieval model. If you construct the opt as in this section, and then pass it in as

default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/mode', override_args=opt)

It should solve your problems

[happysoymilk]

We provide a special script for interacting with the full dialogue retrieval model. If you construct the opt as in this section, and then pass it in as

default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/mode', override_args=opt)

It should solve your problems

Hi, thanks for replying. However, I got the error:

TypeError: create_agent_from_model_file() got an unexpected keyword argument 'override_args'

I checked the parlai official document and there is no this argument for create_agent_from_model_file

[stephenroller]

https://github.com/facebookresearch/ParlAI/blob/d83cb03df6a601f6b8b9387ebfca06c3313994f7/parlai/core/agents.py#L332

it's a typo

[happysoymilk]

https://github.com/facebookresearch/ParlAI/blob/d83cb03df6a601f6b8b9387ebfca06c3313994f7/parlai/core/agents.py#L332

it's a typo

Thank you so much for your answer. However, the agent seems not created:

AttributeError: 'NoneType' object has no attribute 'respond'

I try to print default2.__dir__() and what I got is

['__repr__', '__bool__', '__new__', '__doc__', '__hash__', '__str__', '__getattribute__', '__setattr__', '__delattr__', '__lt__', '__le__', '__eq__', '__ne__', '__gt__', '__ge__', '__init__', '__reduce_ex__', '__reduce__', '__subclasshook__', '__init_subclass__', '__format__', '__sizeof__', '__dir__', '__class__']

[klshuster]

Another typo on my end...

default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/model', opt_overrides=opt)

[happysoymilk]

Another typo on my end...

default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/model', opt_overrides=opt)

Thanks again. However, I encountered new error:

AttributeError: module 'parlai.tasks.wizard_of_wikipedia.agents' has no attribute 'create_agents'

During handling of the above exception, another exception occurred:
...
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\py-anaconda\\envs\\parlai_gpu\\Lib\\site-packages\\data\\wizard_of_wikipedia\\data.json'

[klshuster]

try parlai display_data wizard_of_wikipedia to download the WoW dataset

[happysoymilk]

try parlai display_data wizard_of_wikipedia to download the WoW dataset

Hi, thank you so much again for answering all the questions. I tried to download the whole dataset and the agent should be created for it went to creating task(s): wizard_of_wikipedia. However, there is a new error during the stage:

Traceback (most recent call last):
  File "agenttest.py", line 71, in <module>
    main()
  File "agenttest.py", line 41, in main
    default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/model', opt_overrides=opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 347, in create_agent_from_model_file
    return create_agent_from_opt_file(opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 421, in create_agent_from_opt_file
    return model_class(opt_from_file)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\projects\wizard_of_wikipedia\wizard_transformer_ranker\wizard_transformer_ranker.py", line 76, in __init__
    super().__init__(opt, shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 190, in __init__
    super().__init__(opt, shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_agent.py", line 840, in __init__
    self.set_interactive_mode(opt.get('interactive_mode', False), shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 310, in set_interactive_mode
    path = self.get_task_candidates_path()
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 334, in get_task_candidates_path
    build_cands(opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\scripts\build_candidates.py", line 91, in build_cands
    fw.write('\n'.join(cands))
UnicodeEncodeError: 'gbk' codec can't encode character '\xdc' in position 40155: illegal multibyte sequence

I tried to add encoding='utf-8' to fw = open(outfile, 'w'), while the new error I got is

Traceback (most recent call last):
  File "agenttest.py", line 71, in <module>
    main()
  File "agenttest.py", line 41, in main
    default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/model', opt_overrides=opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 347, in create_agent_from_model_file
    return create_agent_from_opt_file(opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 421, in create_agent_from_opt_file
    return model_class(opt_from_file)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\projects\wizard_of_wikipedia\wizard_transformer_ranker\wizard_transformer_ranker.py", line 76, in __init__
    super().__init__(opt, shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 234, in __init__
    self.set_fixed_candidates(shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 939, in set_fixed_candidates
    vecs = self._make_candidate_vecs(cands)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 995, in _make_candidate_vecs
    [cand_vecs], pad_idx=self.NULL_IDX, dtype=cand_vecs[0].dtype
IndexError: list index out of range

[mojtaba-komeili]

One reason I have seen ParlAI not being able to open the files was if they were damaged or not fully downloaded. Could you try deleting the data and trying again? Data should be in ParlAI/data/wizard_of_wikipedia. Just rm -rf <wiki data dir> and try again.

[happysoymilk]

One reason I have seen ParlAI not being able to open the files was if they were damaged or not fully downloaded. Could you try deleting the data and trying again? Data should be in ParlAI/data/wizard_of_wikipedia. Just rm -rf <wiki data dir> and try again.

Hello, I downloaded the data and tried on the script again, however, the error is still the same:

Traceback (most recent call last):
  File "agenttest.py", line 72, in <module>
    main()
  File "agenttest.py", line 42, in main
    default2 = create_agent_from_model_file(model_file='zoo:wizard_of_wikipedia/full_dialogue_retrieval_model/model', opt_overrides=opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 347, in create_agent_from_model_file
    return create_agent_from_opt_file(opt)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\agents.py", line 421, in create_agent_from_opt_file
    return model_class(opt_from_file)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\projects\wizard_of_wikipedia\wizard_transformer_ranker\wizard_transformer_ranker.py", line 76, in __init__
    super().__init__(opt, shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 234, in __init__
    self.set_fixed_candidates(shared)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 939, in set_fixed_candidates
    vecs = self._make_candidate_vecs(cands)
  File "D:\py-anaconda\envs\parlai_gpu\lib\site-packages\parlai\core\torch_ranker_agent.py", line 995, in _make_candidate_vecs
    [cand_vecs], pad_idx=self.NULL_IDX, dtype=cand_vecs[0].dtype
IndexError: list index out of range

I printed the cand_vecs and the result was [] which means it is not created. I tested the data on end2end Wikipedia model and it worked perfectly with parlai interactive, however parlai interactive cannot be run with the retriever dialogue model.

[stephenroller]

I notice you are using windows, which we don't support officially. (That's why you saw the encoding error)

In this case, it looks like you're supplying data without candidates. Ranking models are different.

[github-actions[bot]]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.