decoding issue

[synetkim]

Hello, I have an issue when I tried to decode the model for TLE.

export THEANO_FLAGS=mode=FAST_RUN,device=gpu3,floatX=float32; $LVSR/exp/wsj/decode_tle.sh wsj_reward6 test 200

Here is the log:

• MODEL=wsj_reward6
• PART=test
• BEAM_SIZE=200
• LM=nolm
• LM_PATH=data/local/nist_lm/wsj_trigram_no_bos
• ls wsj_reward6/reports test_eval92_nolm_200 test_nolm_200
• LM_CONF='monitoring.search.beam_size 200'
• '[' '!' nolm == nolm ']'
• /speech05-ssd/sykim/attention-lvcsr/bin/run.py search --part=test --report wsj_reward6/reports/test_nolm_200 wsj_reward6/annealing1_best_ll.zip /speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml vocabulary data/local/nist_lm/wsj_trigram_no_bos/words.txt monitoring.search.beam_size 200 Using gpu device 3: Tesla K20m (CNMeM is enabled with initial size: 90.0% of memory, cuDNN Version is too old. Update to v5, was 3007.) 2016-06-07 14:49:00,862: pykwalify.core: INFO: validation.valid 2016-06-07 14:49:00,867: pykwalify.core: INFO: validation.valid 2016-06-07 14:49:00,872: pykwalify.core: INFO: validation.valid 2016-06-07 14:49:00,877: pykwalify.core: INFO: validation.valid 2016-06-07 14:49:00,884: main: INFO: Config: {'cmd_args': {'config_changes': <picklable_itertools.extras.equizip object at 0x7fb144a68050>, 'config_path': '/speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml', 'decode_only': None, 'decoded_save': None, 'func': 'search', 'load_path': 'wsj_reward6/annealing1_best_ll.zip', 'nll_only': False, 'params': None, 'part': 'test', 'report': 'wsj_reward6/reports/test_nolm_200', 'seed': 1, 'validate_config': True}, 'data': {'add_bos': 1, 'add_eos': True, 'batch_size': 20, 'dataset_class': <class 'lvsr.datasets.h5py.H5PYAudioDataset'>, 'dataset_filename': 'wsj.h5', 'default_sources': ['recordings', 'labels'], 'name_mapping': {'test': 'test_eval92', 'train': 'train_si284', 'valid': 'test_dev93'}, 'normalization': '', 'sources_map': {'labels': 'characters', 'recordings': 'fbank_dd', 'uttids': 'uttids'}}, 'initialization': {'/recognizer': {'biases_init': Constant(_constant=0.0), 'rec_weights_init': IsotropicGaussian(_mean=0,_std=0.1), 'weights_init': Uniform(_mean=0.0,_width=0.1)}, '/recognizer/generator/readout/post_merge/mlp': {'biases_init': Constant(_constant=-1.0)}}, 'monitoring': {'search': {'beam_size': 200, 'char_discount': 0.0, 'round_to_inf': 4.5, 'stop_on': 'patience'}, 'search_every_batches': 0, 'search_every_epochs': 1, 'validate_every_batches': 0, 'validate_every_epochs': 1}, 'net': {'attention_type': 'content_and_conv', 'bottom': {'activation': <blocks.bricks.simple.Rectifier object at 0x7fb118424510: name=rectifier>, 'bottom_class': <class 'lvsr.bricks.recognizer.SpeechBottom'>, 'dims': []}, 'conv_n': 100, 'criterion': {'min_reward': -5, 'name': 'mse_gain'}, 'dec_transition': <class 'blocks.bricks.recurrent.GatedRecurrent'>, 'dim_dec': 250, 'dims_bidir': [250, 250, 250, 250], 'enc_transition': <class 'blocks.bricks.recurrent.GatedRecurrent'>, 'lm': {'normalize_am_weights': False}, 'max_decoded_length_scale': 3.0, 'post_merge_activation': <blocks.bricks.simple.Rectifier object at 0x7fb1184336d0: name=rectifier>, 'post_merge_dims': [250], 'prior': {'after': 100, 'before': 100, 'initial_begin': 0, 'initial_end': 80, 'max_speed': 4.4, 'min_speed': 2.4, 'type': 'window_around_median'}, 'subsample': [1, 1, 2, 2], 'use_states_for_readout': True}, 'parent': '$LVSR/exp/wsj/configs/wsj_reward4.yaml', 'regularization': {'dropout': False, 'max_norm': 1.0}, 'stages': {'annealing1': {'net': {'lm': None}, 'training': {'epsilon': 1e-10, 'num_epochs': 3, 'restart_from': '_best_ll'}}, 'annealing2': None, 'main': {'net': {'lm': None}, 'training': {'num_epochs': 15, 'restart_from': '_best_ll'}}, 'pretraining': {'net': {'criterion': {'min_reward': -1}, 'lm': None, 'prior': {'initial_begin': 0, 'initial_end': 40, 'max_speed': 2.2, 'min_speed': 1.2, 'type': 'expanding'}}, 'training': {'num_epochs': 4}}}, 'training': {'decay_rate': 0.95, 'epsilon': 1e-08, 'exploration': 'greedy', 'gradient_threshold': 100.0, 'momentum': 0.0, 'rules': ['momentum', 'adadelta'], 'scale': 0.1}, 'vocabulary': 'data/local/nist_lm/wsj_trigram_no_bos/words.txt'} 2016-06-07 14:49:00,996: lvsr.main: INFO: Recognizer initialization started 2016-06-07 14:49:54,017: lvsr.main: INFO: Recognizer is initialized /speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py:493: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 1 is not part of the computational graph needed to compute the outputs: groundtruth. To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'. on_unused_input='warn') Traceback (most recent call last): File "/speech05-ssd/sykim/attention-lvcsr/bin/run.py", line 154, in getattr(lvsr.main, args.pop('func'))(config, args) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 781, in search prediction=raw_groundtruth)[:2] File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 493, in analyze on_unused_input='warn') File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 322, in function output_keys=output_keys) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 480, in pfunc output_keys=output_keys) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1827, in orig_function output_keys=output_keys).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1479, in init accept_inplace) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 177, in std_fgraph update_mapping=update_mapping) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 182, in init self.import_r(output, reason="init") File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 374, in import_r self.import(variable.owner, reason=reason) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 416, in import** variable=r) theano.gof.fg.MissingInputError: An input of the graph, used to compute DimShuffle{0,x}(labels), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.

Backtrace when the variable is created: File "/speech05-ssd/sykim/attention-lvcsr/bin/run.py", line 154, in getattr(lvsr.main, args.pop('func'))(config, _args) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 716, in search recognizer = create_model(config, data, load_path) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 221, in create_model _net_config) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 360, in init self.single_labels = tensor.lvector('labels')

attention-lvcsr/lvsr/main.py", line 716, in search recognizer = create_model(config, data, load_path) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 221, in create_model net_config) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 360, in init** self.single_labels = tensor.lvector('labels')

[dmitriy-serdyuk]

Hmm, that's strange.

Can you run a postmortem debugger and check if input_variables contains the labels (this one)?

[synetkim]

Thanks for replying. Could you explain more detail how to use it?

[dmitriy-serdyuk]

Somehow, a function analyze is not compiled in your case. It just outputs the cost and visualizes the alignment, it is not needed for decoding, but it's very suspicious...

So, I propose to run a debugger like (assuming that you have ipdb; I'm taking the command from your log)

python -mipdb /speech05-ssd/sykim/attention-lvcsr/bin/run.py search --part=test --report wsj_reward6/reports/test_nolm_200 wsj_reward6/annealing1_best_ll.zip /speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml vocabulary data/local/nist_lm/wsj_trigram_no_bos/words.txt monitoring.search.beam_size 200

You can use ... -mpdb ... if you don't want to install ipdb. If you are not familiar with pdb/ipdb, there are a lot of tutorials in internet how to use it.

You'll be able to go up/down the stack and print/inspect the variables.