search

clovaai / FocusSeq2Seq

[EMNLP 2019] Mixture Content Selection for Diverse Sequence Generation (Question Generation / Abstractive Summarization)
https://arxiv.org/abs/1909.01953
MIT License
113 stars 20 forks source link

Getting error in line 237 (of file: QG_data_loader.py) in function: preprocess_single_example #2

Closed riturajkunwar closed 4 years ago

riturajkunwar commented 4 years ago

\QG_data_loader.py", line 237, in preprocess_single_example if word in word2id: NameError: name 'word2id' is not defined

It would be nice if you could help resolve the issue !!

j-min commented 4 years ago

I cannot reproduce the error. Did you run python QG_data_loader.py?

riturajkunwar commented 4 years ago

yes. I am working on Windows machine though. Will this repo work on windows machine ?

Thanks for you response mate !!

j-min commented 4 years ago

word2id is defined here. Please check if your preprocess_single_example function can access this instance.

I don't have a plan for Windows support, since I don't have any Windows machine. Maybe your problem comes from the difference between Windows and Unix path ( '/' and '\'). Try remove './' from these lines and run it again.

riturajkunwar commented 4 years ago

No, it cant access this instance. Is it because "word2id" is defined in "main" and has not been passed as argument to the function "preprocess_single_example" ??

j-min commented 4 years ago

I defined word2id under __main__ to avoid unnecessary pickling during multiprocessing. And instance defined under __main__ is accessible in a function using it. Try run this script. It runs smooth.

def print_word2id():
    print(word2id)

if __name__ == '__main__':
    word2id = 3
    print_word2id()
riturajkunwar commented 4 years ago

Yes you are right. Therefore, "word2id" is accessible in function "preprocess_data" but not in function "preprocess_single_example"

Is it because function "preprocess_single_example" is not called directly called in "main"

j-min commented 4 years ago

It is still accessible. Run this script.

def preprocess_data():
    preprocess_single_example()

def preprocess_single_example():
    print(word2id)

if __name__ == '__main__':
    word2id = 3
    preprocess_data()
riturajkunwar commented 4 years ago

Its weird, as I could print "word2id" in function "preprocess_data" but not in function "preprocess_single_example" .....

I made a DIRTY fix though:

I added the following statement in function "preprocess_single_example":

word2id, id2word = vocab_read(data_dir.joinpath('vocab.txt'))

It generated all the required .pkl files.