Open bhcsayx opened 4 years ago
freesunshine0316
commented
4 years ago Hi Is that the full log? We basically implemented a while loop by ourselves. It seems that this is an out-of-boundary problem when indexing an tensor (indexing 26 from a tensor with shape [8, 26 x]).
bhcsayx
commented
4 years ago Oh sorry I only cut the traceback part… below is the full log I get after running it again
bhcsayx
commented
4 years ago CUDA_VISIBLE_DEVICES 2
Loading the configuration from config.json
Configurations:
<namespace_utils.Bunch object at 0x7f2318ab3790>
Log file path: logs/G2S.cross_bin_0.log
Loading train set.
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_2
Number of training samples: 5313/5313
Number of dev samples: 200/200
Collecting vocabs.
Number of words: 15077
Number of allChars: 137
Number of allEdgelabels: 113
word vocab size 15736
Build DataStream ...
Number of instances in trainDataStream: 5313
Number of instances in devDataStream: 200
Number of batches in trainDataStream: 665
Number of batches in devDataStream: 25
/home/gaoyifei/anaconda3/envs/python27/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2020-04-09 20:52:23.134134: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-09 20:52:25.170465: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[35, :] = [3, 25] does not index into param (shape: [8,25,100]).
2020-04-09 20:52:25.170476: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[35, :] = [3, 25] does not index into param (shape: [8,25,100]).
2020-04-09 20:52:25.184736: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[60, :] = [3, 50] does not index into param (shape: [8,50,100]).
2020-04-09 20:52:25.184743: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[60, :] = [3, 50] does not index into param (shape: [8,50,100]).
Return from here, just evaluate
Start the training loop.
Traceback (most recent call last):
File "G2S_trainer.py", line 307, in
Caused by op u'Train/Model/rev_encoder/gated_operations/while/GatherNd', defined at:
File "G2S_trainer.py", line 307, in
InvalidArgumentError (see above for traceback): flat indices[35, :] = [3, 25] does not index into param (shape: [8,25,100]). [[Node: Train/Model/rev_encoder/gated_operations/while/GatherNd = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Train...ul_1/f_acc"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Train/Model/rev_encoder/gated_operations/while/Identity, Train/Model/rev_encoder/gated_operations/while/stack_1)]]
freesunshine0316
commented
4 years ago From your log, the model crashes once being executed. We didn't experience any of such situation before. Did you change any part of the code?
freesunshine0316
commented
4 years ago @bhcsayx
I'm pretty busy and you don't provide enough information for what has been down, so all I can help is giving you a brief tour on the code does. Hope this can help you a little bit.
Each graph is organized as a list of nodes (1D) and a list of corresponding neighbors (2D). The node is ordered in a topological way, so that it's guaranteed to have processed all the preceding nodes when processing each node.
The while loop iterates left to right on a batch of instances, it first obtains a vertical slice of nodes and their corresponding neighbors, before updating the hidden states for the nodes.
I'm curious how this out-of-boundary indexing happens, which always exceeds by 1 index (e.g. visiting index 25 for a sequence of 25 elements, or visiting index 34 for a sequence of 34 elements). I highly suspect that you made some changes for the datastream part.
juliachen123
commented
3 years ago @freesunshine0316
Hi, thank you for sharing the repo. I've also encountered the same error for flat indices and I didn't change any of the datastream part. Here's the traceback:
Start the training loop.
0 2021-02-17 13:16:26.152175: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
2021-02-17 13:16:26.152176: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
2021-02-17 13:16:26.183353: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[50, :] = [5, 73] does not index into param (shape: [8,73,100]).
2021-02-17 13:16:26.183357: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[50, :] = [5, 73] does not index into param (shape: [8,73,100]).
Traceback (most recent call last):
File "G2S_trainer.py", line 307, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 232, in main
_, loss_value, _ = train_graph.execute(sess, cur_batch, cur_batch_rev, FLAGS, is_train=True)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/G2S_model_graph.py", line 193, in execute
return sess.run([self.accu, self.loss, self.train_op], feed_dict)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
[[Node: Train/Model/rev_encoder/gated_operations/while/GatherNd_1 = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Train...ul_1/f_acc"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Train/Model/rev_encoder/gated_operations/while/Switch_1:1, Train/Model/rev_encoder/gated_operations/while/stack_1)]]
Caused by op u'Train/Model/rev_encoder/gated_operations/while/GatherNd_1', defined at:
File "G2S_trainer.py", line 307, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 186, in main
char_vocab=char_vocab, options=FLAGS, mode='train')
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/G2S_model_graph.py", line 62, in __init__
is_training = is_training, options = options)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/dag_encoder_utils.py", line 276, in __init__
idx_var.get_shape(),])
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3232, in while_loop
return_same_structure)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2952, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2887, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/dag_encoder_utils.py", line 229, in _recurrence
prev_cell = collect_neighbor_node_representations_2D(passage_node_cell, prev_idx)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/dag_encoder_utils.py", line 14, in collect_neighbor_node_representations_2D
return tf.gather_nd(representation, indices)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3052, in gather_nd
"GatherNd", params=params, indices=indices, name=name)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
[[Node: Train/Model/rev_encoder/gated_operations/while/GatherNd_1 = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Train...ul_1/f_acc"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Train/Model/rev_encoder/gated_operations/while/Switch_1:1, Train/Model/rev_encoder/gated_operations/while/stack_1)]]```
Any suggestions for where to look into? Thank you in advance!
mank-hub
commented
3 years ago The error(s) are because of the mismatching in the config.json file : For the default configurations ["word_format": "lemma" and "class_num": 2,], the gs_lstm model works fine.
Hi, I am trying to run your implementation of Peng's model, and I came across with the problem of the wrong input indices, shown as the image below, which made me confused. Do you have any ideas? Thank you in advance!