search

bloomsburyai / question-generation

Neural text-to-text question generation
MIT License
217 stars 52 forks source link

error when running eval.py #6

Closed rpoli40 closed 5 years ago

rpoli40 commented 5 years ago

Used commit recommended (be13417) and tried to run python eval.py --data_path ./data/ Got the following error: sh-4.2$ python eval.py --data_path ./data/ WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives. Traceback (most recent call last): File "eval.py", line 265, in tf.app.run() File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run _sys.exit(main(argv)) File "eval.py", line 41, in main dev_data = loader.load_squad_triples(FLAGS.data_path, dev=FLAGS.eval_on_dev, test=FLAGS.eval_on_test) File "/home/ec2-user/SageMaker/question-generation-be134175652204f3bf51cb194454d7b72c8b8105/src/helpers/loader.py", line 32, in load_squad_triples raw_data = load_squad_dataset(path, dev=dev, test=test, v2=v2) File "/home/ec2-user/SageMaker/question-generation-be134175652204f3bf51cb194454d7b72c8b8105/src/helpers/loader.py", line 23, in load_squad_dataset with open(path+filename) as dataset_file: FileNotFoundError: [Errno 2] No such file or directory: './data/dev-v1.1.json' sh-4.2$ python eval.py --data_path ../data/ WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives. ERROR Eval dataset is smaller than the num_eval_samples flag! sh-4.2$

How can I run the demo without using flask app? Thank you

tomhosking commented 5 years ago

FileNotFoundError: [Errno 2] No such file or directory: './data/dev-v1.1.json'

This is the important part. Make sure you have a dataset in the data directory. Either use the official SQuAD split, or this repo containing the split from Du et al 2017. Then, use --noeval_on_dev --eval_on_test to evaluate on the test set.

The demo is built in flask so you need flask to run the full demo.

rpoli40 commented 5 years ago

Sorry, copied two errors instead only the last one. . The error of data was resolved. File dev-v1.1.json was used from repo be13417, however got the following error: ERROR Eval dataset is smaller than the num_eval_samples flag!

tomhosking commented 5 years ago

The files from the repo are based on my own split - they contain 4691 samples for dev or 5609 for test, you'll need to set the --num_dev_samples or --num_eval_samples flag accordingly.

rpoli40 commented 5 years ago

Looks like that repo is missing some files (this folder models/saved...doesn't exist): File "/home/ec2-user/SageMaker/question-generation-be134175652204f3bf51cb194454d7b72c8b8105/src/langmodel/lm.py", line 101,in load_from_chkpt with open(path+'/vocab.json') as f: FileNotFoundError: [Errno 2] No such file or directory: '../models/saved/lmtest/vocab.json' Exception ignored in: <bound method LstmLmInstance.del of <langmodel.lm.LstmLmInstance object at 0x7f9d8b8cb550>> Traceback (most recent call last): File "/home/ec2-user/SageMaker/question-generation-be134175652204f3bf51cb194454d7b72c8b8105/src/langmodel/lm.py", line 98, in del self.sess.close() AttributeError: 'LstmLmInstance' object has no attribute 'sess' Exception ignored in: <bound method QANetInstance.del of <qa.qanet.instance.QANetInstance object at 0x7f9d8b8cb518>> Traceback (most recent call last): File "/home/ec2-user/SageMaker/question-generation-be134175652204f3bf51cb194454d7b72c8b8105/src/qa/qanet/instance.py", line53, in del self.sess.close() AttributeError: 'QANetInstance' object has no attribute 'sess'

tomhosking commented 5 years ago

That repo contains a complete dataset - but eval.py uses a language model and QA model to score the generated questions, which you're missing (and I haven't released - I forgot they were required). The demo doesn't need these.

shizhediao commented 5 years ago

That repo contains a complete dataset - but eval.py uses a language model and QA model to score the generated questions, which you're missing (and I haven't released - I forgot they were required). The demo doesn't need these.

Could you upload the missing files? I'm trying to evaluate by running eval.py and found that I need these files. Thanks!