search

julianser / hed-dlg-truncated

Hierarchical Encoder Decoder RNN (HRED) with Truncated Backpropagation Through Time (Truncated BPTT)
GNU General Public License v3.0
308 stars 129 forks source link

InconsistencyError: Trying to reintroduce a removed node #28

Open dhirajmadan1 opened 6 years ago

dhirajmadan1 commented 6 years ago

I am facing error 'InconsistencyError: Trying to reintroduce a removed node' while running HRED on ubuntu dataset. Here is the stack trace:- ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x2b5b16dd7b50> 2017-11-14 01:11:53,554: theano.gof.opt: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x2b5b16dd7b50> ERROR (theano.gof.opt): Traceback: 2017-11-14 01:11:53,554: theano.gof.opt: ERROR: Traceback: ERROR (theano.gof.opt): Traceback (most recent call last): File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply sub_prof = optimizer.optimize(fgraph) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/opt.py", line 87, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 391, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 365, in replace_allvalidate fgraph.validate() File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 256, in validate ret = fgraph.execute_callbacks('validate') File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/fg.py", line 589, in execute_callbacks fn(self, args, **kwargs) File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 422, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") InconsistencyError: Trying to reintroduce a removed node

I am calling it as:- THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python train.py --prototype prototype_ubuntu_HRED

Thanks and Regards Dhiraj

cgsdfc commented 5 years ago

I got the same error, but the training script didn't stop for that. Command I used: ssh://cgsdfc@192.168.1.175:22:docker run --runtime nvidia -v /home/cgsdfc/deployment/HRED-VHRED:/root/HRED-VHRED -v /home/cgsdfc/UbuntuDialogueCorpus:/home/cgsdfc/UbuntuDialogueCorpus -w /root/HRED-VHRED -e PYTHONPATH=/root/HRED-VHRED ufoym/deepo:theano-py36-cu90 python bin/train.py prototype_ubuntu_HRED

error log: 2019-04-02 06:08:12,502: serban.dialog_encoder_decoder:1420: DEBUG: Building train function /root/HRED-VHRED/serban/dialog_encoder_decoder.py:1430: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 1 is not part of the computational graph needed to compute the outputs: x_data_reversed. To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'. name="train_fn") /root/HRED-VHRED/serban/dialog_encoder_decoder.py:1430: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 5 is not part of the computational graph needed to compute the outputs: ran_cost_utterance. To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'. name="train_fn") /root/HRED-VHRED/serban/dialog_encoder_decoder.py:1430: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 6 is not part of the computational graph needed to compute the outputs: x_dropmask. To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'. name="train_fn") ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240> 2019-04-02 06:08:13,636: theano.gof.opt:198: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240> ERROR (theano.gof.opt): Traceback: 2019-04-02 06:08:13,636: theano.gof.opt:199: ERROR: Traceback: ERROR (theano.gof.opt): Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_allvalidate fgraph.validate() File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks fn(self, args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

2019-04-02 06:08:13,637: theano.gof.opt:200: ERROR: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_allvalidate fgraph.validate() File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks fn(self, args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240> 2019-04-02 06:08:14,208: theano.gof.opt:198: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240> ERROR (theano.gof.opt): Traceback: 2019-04-02 06:08:14,208: theano.gof.opt:199: ERROR: Traceback: ERROR (theano.gof.opt): Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_allvalidate fgraph.validate() File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks fn(self, args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

2019-04-02 06:08:14,208: theano.gof.opt:200: ERROR: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_allvalidate fgraph.validate() File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks fn(self, args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240> 2019-04-02 06:08:14,502: theano.gof.opt:198: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240> ERROR (theano.gof.opt): Traceback: 2019-04-02 06:08:14,502: theano.gof.opt:199: ERROR: Traceback: ERROR (theano.gof.opt): Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_allvalidate fgraph.validate() File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks fn(self, args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

2019-04-02 06:08:14,503: theano.gof.opt:200: ERROR: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize ret = self.apply(fgraph, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply node = self.process_node(fgraph, node) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node node, args) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars add_as_nitsots) File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs reason='scanOp_pushout_output') File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove chk = fgraph.replace_all_validate(replacements, reason) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_allvalidate fgraph.validate() File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks fn(self, args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate raise theano.gof.InconsistencyError("Trying to reintroduce a removed node") theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

If the training don't stop for this error, can I just ignore it? Thanks and Regards Cong