I am using script base_size.sh to run the class run_classifier.py. I am able to train and evaluate on imdb data for attention_type set as _originalfull and _blocksparse but when I set it to _simulatedsparse I see errors in initializing the training itself. The 12 layers are initialized but training doesn't start. The major error log is below:
File "/home/amitghattimare/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3211, in _as_graph_def
graph.ParseFromString(compat.as_bytes(data))
google.protobuf.message.DecodeError: Error parsing message
I used the below script to run the code in case it helps in investigation. If I change attention_type to the other 2 options, it works fine. I am using only 8 cores because that's the max available in preemptible mode. I have reduced train_batch_size so that it fits in memory. I wonder if that's causing the issue though error logs don't indicate that.
I am using script
base_size.sh
to run the classrun_classifier.py
. I am able to train and evaluate on imdb data forattention_type
set as _originalfull and _blocksparse but when I set it to _simulatedsparse I see errors in initializing the training itself. The 12 layers are initialized but training doesn't start. The major error log is below:I used the below script to run the code in case it helps in investigation. If I change attention_type to the other 2 options, it works fine. I am using only 8 cores because that's the max available in preemptible mode. I have reduced train_batch_size so that it fits in memory. I wonder if that's causing the issue though error logs don't indicate that.