Error while compiling the model after loading the google-BERT model on tpu

[adnan33]

I was running the tutorial notebook on google colab and faced this issue.

Code

# @title Compile keras  model here
from transformer.train import train_model
if use_tpu:
  assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; Maybe you should switch hardware accelerator to TPU for TPU support'
  import tensorflow as tf
  tpu_address = 'grpc://' + os.environ['COLAB_TPU_ADDR']
  strategy = tf.contrib.tpu.TPUDistributionStrategy(
          tf.contrib.cluster_resolver.TPUClusterResolver(tpu=tpu_address)
  )
  g_bert = tf.contrib.tpu.keras_to_tpu_model(
                      g_bert, strategy=strategy)
g_bert.compile('adam', 'mse')  

Error

InvalidArgumentError                      Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1334     try:
-> 1335       return fn(*args)
   1336     except errors.OpError as e:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1317       # Ensure any changes to the graph are reflected in the runtime.
-> 1318       self._extend_graph()
   1319       return self._call_tf_sessionrun(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _extend_graph(self)
   1352     with self._graph._session_run_lock():  # pylint: disable=protected-access
-> 1353       tf_session.ExtendSession(self._session)
   1354 

InvalidArgumentError: NodeDef mentions attr 'explicit_paddings' not in Op<name=Conv2D; signature=input:T, filter:T -> output:T; attr=T:type,allowed=[DT_HALF, DT_BFLOAT16, DT_FLOAT, DT_DOUBLE]; attr=strides:list(int); attr=use_cudnn_on_gpu:bool,default=true; attr=padding:string,allowed=["SAME", "VALID"]; attr=data_format:string,default="NHWC",allowed=["NHWC", "NCHW"]; attr=dilations:list(int),default=[1, 1, 1, 1]>; NodeDef: {{node layer_0/c_attn/conv1d}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-9-8db02f074733> in <module>()
      9   g_bert = tf.contrib.tpu.keras_to_tpu_model(
     10                       g_bert, strategy=strategy)
---> 11 g_bert.compile('adam', 'mse')

/content/bert_keras_repo/transformer/__init__.py in tpu_compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, **kwargs)
     36                     sample_weight_mode, weighted_metrics,
     37                     target_tensors, **kwargs)
---> 38         initialize_uninitialized_variables() # for unknown reason, we should run this after compile sometimes
     39     KerasTPUModel.compile = tpu_compile
     40 

/content/bert_keras_repo/transformer/__init__.py in initialize_uninitialized_variables()
     15     from tensorflow.contrib.tpu.python.tpu.keras_support import KerasTPUModel
     16     def initialize_uninitialized_variables():
---> 17         sess = K.get_session()
     18         uninitialized_variables = set([i.decode('ascii') for i in sess.run(tf.report_uninitialized_variables())])
     19         init_op = tf.variables_initializer(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py in get_session()
    430   if not _MANUAL_VAR_INIT:
    431     with session.graph.as_default():
--> 432       _initialize_variables(session)
    433   return session
    434 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py in _initialize_variables(session)
    706     # marked as initialized.
    707     is_initialized = session.run(
--> 708         [variables_module.is_variable_initialized(v) for v in candidate_vars])
    709     uninitialized_vars = []
    710     for flag, v in zip(is_initialized, candidate_vars):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    928     try:
    929       result = self._run(None, fetches, feed_dict, options_ptr,
--> 930                          run_metadata_ptr)
    931       if run_metadata:
    932         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1151     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1152       results = self._do_run(handle, final_targets, final_fetches,
-> 1153                              feed_dict_tensor, options, run_metadata)
   1154     else:
   1155       results = []

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1327     if handle is None:
   1328       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1329                            run_metadata)
   1330     else:
   1331       return self._do_call(_prun_fn, handle, feeds, fetches)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1347           pass
   1348       message = error_interpolation.interpolate(message, self._graph)
-> 1349       raise type(e)(node_def, op, message)
   1350 
   1351   def _extend_graph(self):

InvalidArgumentError: NodeDef mentions attr 'explicit_paddings' not in Op<name=Conv2D; signature=input:T, filter:T -> output:T; attr=T:type,allowed=[DT_HALF, DT_BFLOAT16, DT_FLOAT, DT_DOUBLE]; attr=strides:list(int); attr=use_cudnn_on_gpu:bool,default=true; attr=padding:string,allowed=["SAME", "VALID"]; attr=data_format:string,default="NHWC",allowed=["NHWC", "NCHW"]; attr=dilations:list(int),default=[1, 1, 1, 1]>; NodeDef: node layer_0/c_attn/conv1d (defined at bert_keras_repo/transformer/model.py:20) . (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).

Errors may have originated from an input operation.
Input Source operations connected to node layer_0/c_attn/conv1d:
 layer_normalization/gamma (defined at bert_keras_repo/transformer/layers.py:45)    
 layer_normalization/beta (defined at bert_keras_repo/transformer/layers.py:46) 
 layer_normalization/Mean/reduction_indices (defined at bert_keras_repo/transformer/layers.py:50)   
 layer_normalization/add/y (defined at bert_keras_repo/transformer/layers.py:52)    
 PositionEmbedding/embeddings (defined at bert_keras_repo/transformer/embedding.py:67)  
 token_input (defined at bert_keras_repo/transformer/model.py:63)   
 position_input (defined at bert_keras_repo/transformer/model.py:65)    
 segment_input (defined at bert_keras_repo/transformer/model.py:64) 
 TokenEmbedding/embeddings (defined at bert_keras_repo/transformer/embedding.py:68) 
 layer_normalization/Mean_1/reduction_indices (defined at bert_keras_repo/transformer/layers.py:51) 
 keras_learning_phase/input (defined at bert_keras_repo/transformer/embedding.py:73)    
 SegmentEmbedding/embeddings (defined at bert_keras_repo/transformer/embedding.py:66)

[Separius]

Well, I searched for the error and it seems that it's a versioning issue, the pretrained model was created with an older version of TF than the one you are using right now.

Can you downgrade your TF and let me how it goes?

[adnan33]

Well, I searched for the error and it seems that it's a versioning issue, the pretrained model was created with an older version of TF than the one you are using right now.

Can you downgrade your TF and let me how it goes? Thank you for the quick reply. Downgrading to tensorflow 1.13.0rc0 solves the issue. Thanks again.