search

rkadlec / asreader

This is an implementation of the Attention Sum Reader model as presented in "Text Comprehension with the Attention Sum Reader Network" available at http://arxiv.org/abs/1603.01547.
Other
95 stars 32 forks source link

fuel.exceptions.ConfigurationError: Configuration not set and no default provided: floatX. #14

Open wenwei202 opened 7 years ago

wenwei202 commented 7 years ago

By running ./quick-start-cbt-ne.sh, what might be the issue?

systems Ubuntu 16.04 with python and theano of

$ python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import theano as th
Using cuDNN version 5110 on context None
Mapped name None to device cuda: TITAN X (Pascal) (0000:22:00.0)
>>> th.__version__
'0.9.0.dev-c697eeab84e5b8a74908da654b66ec9eca4f1291'
>>> 

$ ./quick-start-cbt-ne.sh 
+++ DATA_ROOT=../data/CBTest/data/
+++ TRAIN=cbtest_NE_train.txt
+++ VALID=cbtest_NE_valid_2000ex.txt
+++ TEST=cbtest_NE_test_2500ex.txt
+++ OUT_DIR=test_output_cbt
+++ DATASET_TYPE=cbt
+++ for HYPERPARAMETERS in '"${@:7}"'
+++ LOG_FILE=cbtest_NE_train.txt.-ehd_128_-sed_256.log
+++ mkdir test_output_cbt -p
+++ THEANO_FLAGS=floatX=float32,device=gpu
+++ PYTHONPATH=~/github/::./quick-start-generic.sh
+++ python text_comprehension/as_reader.py -b 32 -ehd 128 -sed 256 -lr 0.001 -p 2 --dataset_root ../data/CBTest/data/ --train cbtest_NE_train.txt --valid cbtest_NE_valid_2000ex.txt --test cbtest_NE_test_2500ex.txt --output_dir test_output_cbt --evaluate_every_n 1e --patience_metric accuracy --own_eval --append_metaparams --no_html --dataset_type cbt --save_path test_output_cbt/model.blocks.pkl
+++ tee test_output_cbt/cbtest_NE_train.txt.-ehd_128_-sed_256.log -i
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: TITAN X (Pascal) (CNMeM is disabled, cuDNN 5110)
Traceback (most recent call last):
  File "text_comprehension/as_reader.py", line 166, in <module>
Output will be stored in test_output_cbt
Computing new vocabulary for file ../data/CBTest/data/cbtest_NE_train.txt.
Processed line 100000
Processed line 200000
Processed line 300000
Processed line 400000
Processed line 500000
Processed line 600000
Processed line 700000
Processed line 800000
Processed line 900000
Processed line 1000000
Processed line 1100000
Processed line 1200000
Processed line 1300000
Processed line 1400000
Processed line 1500000
Processed line 1600000
Processed line 1700000
Processed line 1800000
Processed line 1900000
Processed line 2000000
Processed line 2100000
Processed line 2200000
Processed line 2300000
STATISTICS
Total words: 53825761
Total distinct words: 60278
    exp.execute()
  File "~/github/asreader/asreader/text_comprehension/text_comprehension_base.py", line 343, in execute
    read_ahead=self.args.sort_k_batches_ahead)
  File "~/github/asreader/asreader/text_comprehension/text_comprehension_base.py", line 307, in get_stream
    stream = Padding(stream, mask_sources=['context', 'question', 'candidates'])
  File "~/.local/lib/python2.7/site-packages/fuel/transformers/__init__.py", line 703, in __init__
    self.mask_dtype = config.floatX
  File "~/.local/lib/python2.7/site-packages/fuel/config_parser.py", line 150, in __getattr__
    "provided: {}.".format(key))
fuel.exceptions.ConfigurationError: Configuration not set and no default provided: floatX.
+++ for HYPERPARAMETERS in '"${@:7}"'
+++ LOG_FILE=cbtest_NE_train.txt.-ehd_384_-sed_384.log
+++ mkdir test_output_cbt -p
+++ THEANO_FLAGS=floatX=float32,device=gpu
+++ tee test_output_cbt/cbtest_NE_train.txt.-ehd_384_-sed_384.log -i
+++ PYTHONPATH=/tmp::./quick-start-generic.sh
+++ python text_comprehension/as_reader.py -b 32 -ehd 384 -sed 384 -lr 0.001 -p 2 --dataset_root ../data/CBTest/data/ --train cbtest_NE_train.txt --valid cbtest_NE_valid_2000ex.txt --test cbtest_NE_test_2500ex.txt --output_dir test_output_cbt --evaluate_every_n 1e --patience_metric accuracy --own_eval --append_metaparams --no_html --dataset_type cbt --save_path test_output_cbt/model.blocks.pkl
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: TITAN X (Pascal) (CNMeM is disabled, cuDNN 5110)
Traceback (most recent call last):
  File "text_comprehension/as_reader.py", line 166, in <module>
    Output will be stored in test_output_cbt
Computing new vocabulary for file ../data/CBTest/data/cbtest_NE_train.txt.
Processed line 100000
Processed line 200000
Processed line 300000
Processed line 400000
Processed line 500000
Processed line 600000
Processed line 700000
Processed line 800000
Processed line 900000
Processed line 1000000
Processed line 1100000
Processed line 1200000
Processed line 1300000
Processed line 1400000
Processed line 1500000
Processed line 1600000
Processed line 1700000
Processed line 1800000
Processed line 1900000
Processed line 2000000
Processed line 2100000
Processed line 2200000
Processed line 2300000
STATISTICS
Total words: 53825761
Total distinct words: 60278
exp.execute()
  File "~/github/asreader/asreader/text_comprehension/text_comprehension_base.py", line 343, in execute
    read_ahead=self.args.sort_k_batches_ahead)
  File "~/github/asreader/asreader/text_comprehension/text_comprehension_base.py", line 307, in get_stream
    stream = Padding(stream, mask_sources=['context', 'question', 'candidates'])
  File "~/.local/lib/python2.7/site-packages/fuel/transformers/__init__.py", line 703, in __init__
    self.mask_dtype = config.floatX
  File "~/.local/lib/python2.7/site-packages/fuel/config_parser.py", line 150, in __getattr__
    "provided: {}.".format(key))
fuel.exceptions.ConfigurationError: Configuration not set and no default provided: floatX.
+++ python text_comprehension/eval/copyBestPredictions.py -vp validation.txt. -tp test.txt. -i test_output_cbt -o test_output_cbt/best_predictions
Validation files:

Best predictions to be copied:

Best validation model:
Traceback (most recent call last):
  File "text_comprehension/eval/copyBestPredictions.py", line 124, in <module>
    print bestValModel['params']
KeyError: 'params'
wenwei202 commented 7 years ago

I can skip this error by adding floatX: float32 in ~/.fuelrc but get another issue:

$ ./quick-start-cbt-ne.sh 
+++ DATA_ROOT=../data/CBTest/data/
+++ TRAIN=cbtest_NE_train.txt
+++ VALID=cbtest_NE_valid_2000ex.txt
+++ TEST=cbtest_NE_test_2500ex.txt
+++ OUT_DIR=test_output_cbt
+++ DATASET_TYPE=cbt
+++ for HYPERPARAMETERS in '"${@:7}"'
+++ LOG_FILE=cbtest_NE_train.txt.-ehd_128_-sed_256.log
+++ mkdir test_output_cbt -p
+++ THEANO_FLAGS=floatX=float32,device=gpu
+++ PYTHONPATH=/tmp::./quick-start-generic.sh
+++ python text_comprehension/as_reader.py -b 32 -ehd 128 -sed 256 -lr 0.001 -p 2 --dataset_root ../data/CBTest/data/ --train cbtest_NE_train.txt --valid cbtest_NE_valid_2000ex.txt --test cbtest_NE_test_2500ex.txt --output_dir test_output_cbt --evaluate_every_n 1e --patience_metric accuracy --own_eval --append_metaparams --no_html --dataset_type cbt --save_path test_output_cbt/model.blocks.pkl
+++ tee test_output_cbt/cbtest_NE_train.txt.-ehd_128_-sed_256.log -i
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: TITAN X (Pascal) (CNMeM is disabled, cuDNN 5110)
ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7f05131dda50>
ERROR:theano.gof.opt:SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7f05131dda50>
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 87, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "~/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 685, in apply
    node = self.process_node(fgraph, node)
  File "~/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 745, in process_node
    node, args)
  File "~/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
    add_as_nitsots)
  File "~/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
    reason='scanOp_pushout_output')
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 391, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 365, in replace_all_validate
    fgraph.validate()
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 256, in validate_
    ret = fgraph.execute_callbacks('validate')
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/fg.py", line 589, in execute_callbacks
    fn(self, *args, **kwargs)
  File "~/anaconda2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 422, in validate
    raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
InconsistencyError: Trying to reintroduce a removed node
wenwei202 commented 7 years ago

skip raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") by deleting THEANO_FLAGS in quick-start-generic.sh, and using default ones in ~/.theanorc.

#~/.theanorc
[global]
device = cuda
floatX = float32
force_device = True

But got new issue:

$ ./quick-start-cbt-ne.sh
+++ DATA_ROOT=../data/CBTest/data/
+++ TRAIN=cbtest_NE_train.txt
+++ VALID=cbtest_NE_valid_2000ex.txt
+++ TEST=cbtest_NE_test_2500ex.txt
+++ OUT_DIR=test_output_cbt
+++ DATASET_TYPE=cbt
+++ for HYPERPARAMETERS in '"${@:7}"'
+++ LOG_FILE=cbtest_NE_train.txt.-ehd_128_-sed_256.log
+++ mkdir test_output_cbt -p
+++ PYTHONPATH=/tmp::./quick-start-generic.sh
+++ python text_comprehension/as_reader.py -b 32 -ehd 128 -sed 256 -lr 0.001 -p 2 --dataset_root ../data/CBTest/data/ --train cbtest_NE_train.txt --valid cbtest_NE_valid_2000ex.txt --test cbtest_NE_test_2500ex.txt --output_dir test_output_cbt --evaluate_every_n 1e --patience_metric accuracy --own_eval --append_metaparams --no_html --dataset_type cbt --save_path test_output_cbt/model.blocks.pkl
+++ tee test_output_cbt/cbtest_NE_train.txt.-ehd_128_-sed_256.log -i
Using cuDNN version 5110 on context None
Mapped name None to device cuda: TITAN X (Pascal) (0000:22:00.0)
Traceback (most recent call last):
  File "text_comprehension/as_reader.py", line 166, in <module>
Output will be stored in test_output_cbt
Computing new vocabulary for file ../data/CBTest/data/cbtest_NE_train.txt.
Processed line 100000
Processed line 200000
Processed line 300000
Processed line 400000
Processed line 500000
Processed line 600000
Processed line 700000
Processed line 800000
Processed line 900000
Processed line 1000000
Processed line 1100000
Processed line 1200000
Processed line 1300000
Processed line 1400000
Processed line 1500000
Processed line 1600000
Processed line 1700000
Processed line 1800000
Processed line 1900000
Processed line 2000000
Processed line 2100000
Processed line 2200000
Processed line 2300000
STATISTICS
Total words: 53825761
Total distinct words: 60278
STATISTICS
Total words: 942535
Total distinct words: 10720

Added 1311 new words from file ../data/CBTest/data/cbtest_NE_test_2500ex.txt to previous vocabulary.
    exp.execute()
  File "/home//github/asreader/asreader/text_comprehension/text_comprehension_base.py", line 386, in execute
    **self.args.evaluate_every_n)
  File "/home//github/asreader/asreader/text_comprehension/monitoring.py", line 79, in __init__
    self._evaluator = MemoryDatasetEvaluator([context_attention, context,y_hat,y,candidates, candidates_mask, context_mask,x_mask,x], updates)
  File "/home//github/asreader/asreader/text_comprehension/monitoring.py", line 237, in __init__
    self._compile()
  File "/home//github/asreader/asreader/text_comprehension/monitoring.py", line 277, in _compile
    outputs)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/compile/function.py", line 326, in function
    output_keys=output_keys)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1794, in orig_function
    output_keys=output_keys).create(
  File "/home//anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1474, in __init__
    optimizer_profile = optimizer(fgraph)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 98, in __call__
    return self.optimize(fgraph)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 87, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 87, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/opt.py", line 87, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gpuarray/opt.py", line 382, in apply
    node.outputs)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gpuarray/opt.py", line 1189, in local_gpua_gemmbatch
    return gpugemmbatch_no_inplace(c, 1.0, a, b, 0.0)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gof/op.py", line 615, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/home//anaconda2/lib/python2.7/site-packages/theano/gpuarray/blas.py", line 391, in make_node
    assert A.ndim == 3
AssertionError
rkadlec commented 7 years ago

I have never seen something like that. Do you use the version of Blocks mentioned in readme.md? We also used older version of Theano than 0.9.0 as indicated in our logs. Let me know whether it helped.

mrcocytus commented 7 years ago

The same issue as above. The dependencies are quite annoying. I guess it can be helpful that the author gives the whole pip list.

Seayoung277 commented 7 years ago

Got exactly the same problems and finally solved by:

  1. Find the same commit version of Blocks. Actually I looked through all the commit sshs to find the desired one and finally downloaded the zip for installation.
  2. Install Theano 0.8.0 instead of 0.9.0 before installing Blocks. If you install the old version Blocks first without Theano, it will install the latest verson Theano as well as numpy-1.9.3. You also have to reinstall numpy in order to correctly run the asreader. Older versions such as 1.9.3 or latest versions such as 1.12.1 can lead to different problems. I tried numpy-1.10.0 and it worked. I haven't tried other versions yet.

Currently the code is still running on my computer. I don't have a GPU so it'll take a long time. Till now everything seems alright.

byplusone commented 7 years ago

@seayoungzhang Did your code run successfully? I tried but still meet this problem. Can you show me your pip list? Thanks!

Seayoung277 commented 7 years ago

ALL PROBLEMS SOLVED

After following the steps presented above you may end in the error issued in #12 Please refer to the solution there.

Seayoung277 commented 7 years ago

And here's my pip list:

adium-theme-ubuntu (0.3.4) alabaster (0.7.10) Babel (2.4.0) bleach (1.5.0) blocks (0.2.0, /home/seayoung/blocks) docutils (0.13.1) fuel (0.2.0) funcsigs (1.0.2) h5py (2.7.0) html5lib (0.9999999) imagesize (0.7.1) Jinja2 (2.9.6) Markdown (2.2.0) MarkupSafe (1.0) mock (2.0.0) nltk (3.2.4) nose (1.3.7) nose2 (0.6.5) numexpr (2.6.2) numpy (1.10.0) olefile (0.44) pbr (3.0.1) picklable-itertools (0.1.1) Pillow (4.1.1) pip (9.0.1) progressbar2 (3.6.0) protobuf (3.3.0) Pygments (2.2.0) python-apt (1.1.0b1) pytz (2017.2) PyYAML (3.11) pyzmq (16.0.2) requests (2.14.2) scipy (0.19.0) setuptools (20.7.0) six (1.10.0) snowballstemmer (1.2.1) Sphinx (1.6.1) sphinx-rtd-theme (0.2.4) sphinxcontrib-websupport (1.0.1) tables (3.4.2) tensorflow (1.2.0rc0) Theano (0.8.0, /usr/local/lib/python2.7/dist-packages) toolz (0.7.2) tqdm (4.11.2) typing (3.6.1) unity-lens-photos (1.0) Werkzeug (0.12.2) wheel (0.29.0) yadlt (0.0.6)

Some of them are irrelevant to this code. Just for reference.

purn3ndu commented 7 years ago

@seayoungzhang Do you have the commit ssh you used for installing blocks as you mentioned above? Thanks! :)

Seayoung277 commented 7 years ago

@purnendu91 359afad119f8c6ac0ebc3cc6ec6e6475656babae It's provided in the project README file Good Luck ! :)

fiberleif commented 6 years ago

step 1: adding floatX: float32 in ~/.fuelrc step 2: ~/.theanorc [global] device = cuda floatX = float32

it seems work for my task @Actor-Critic for Sequence Prediction

DAHANEYassine commented 6 years ago

someone help me please ConfigurationError: Configuration not set and no default provided: floatX. on windows 10, anaconda3

fmartinssp commented 6 years ago

DAHANEYassine Did you fix the problem? I have the same.

DAHANEYassine commented 6 years ago

fmartnissp i don't figure out how i can solve it.