search

tensorflow / hub

A library for transfer learning by reusing parts of TensorFlow models.
https://tensorflow.org/hub
Apache License 2.0
3.49k stars 1.67k forks source link

USEV4 Tf2 Fine Tuning Generates Large Number of Warnings #446

Closed meethariprasad closed 4 years ago

meethariprasad commented 4 years ago

Hi All,

While fine tuning USE4, in TF2, we get large number of Warnings. My assumption is these can be ignored as they are actually not trainable variables and TF is just letting us know that. Want to reconfirm if this understanding is true and Fine Tuning is infact happening for trainable variables. Code to Reproduce: `

!pip install tensorflow=2.1.0-rc0
import tensorflow as tf
import tensorflow_hub as hub
from tensorflow import keras
print (tf.__version__)

text_values_list=['hive','.net','OBIEE']
labels=[0,1,2]

hub_url='https://tfhub.dev/google/universal-sentence-encoder-large/4'
module_obj=hub.load(hub_url)
hub_layer=hub.KerasLayer(module_obj,trainable=True,input_shape=[], dtype=tf.string)

model = keras.Sequential()
model.add(hub_layer)
model.add(keras.layers.Dense(16, activation='relu'))
model.add(keras.layers.Dense(len(labels), activation='softmax'))
model.compile(optimizer='rmsprop', loss=tf.keras.losses.SparseCategoricalCrossentropy())

model.fit(x=text_values_list,y=labels,epochs=2)
model.summary()`

`WARNING:tensorflow:Gradients do not exist for variables ['EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_29:0'] when minimizing the loss.

WARNING:tensorflow:Gradients do not exist for variables ['EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_1/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_2/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_29:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_0:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_1:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_2:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_3:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_4:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_5:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_6:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_7:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_8:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_9:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_10:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_11:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_12:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_13:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_14:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_15:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_16:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_17:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_18:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_19:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_20:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_21:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_22:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_23:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_24:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_25:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_26:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_27:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_28:0', 'EncoderDNN/DNN/ResidualHidden_3/AdjustDepth/projection/kernel/part_29:0'] when minimizing the loss. `

gowthamkpr commented 4 years ago

I am not getting as many warnings as you are encountering @meethariprasad I am using tf-nightly. Please find my gist here

ARArun commented 4 years ago

@gowthamkpr I think you are also getting similar amount of errors. ie what @meethariprasad has posted is the folded version of the the four warning messages you are getting here.

meethariprasad commented 4 years ago

@gowthamkpr I think you are also getting similar amount of errors. ie what @meethariprasad has posted is the folded version of the the four warning messages you are getting here.

Yes. @gowthamkpr You are also getting warnings as I observed.

ARArun commented 4 years ago

@meethariprasad @gowthamkpr These warnings tell us gradients not present for almost all layers.

What does this warning mean.

Is fine tuning happening or not.

meethariprasad commented 4 years ago

@ARArun I don't think that represent all layers. We have more than 10,00,000,00 parameters in USE embeddings and all variables are not trainable. My assumption is TF is showing the ones with frozen variables that can't be trained.

I can't say it for sure. TF team need to confirm.

ARArun commented 4 years ago

@meethariprasad Thanks for the clarification.

gowthamkpr commented 4 years ago

@meethariprasad Yes you are right. You will get gradients only for the trainable variables. As the layers have frozen variables that cant be trained hence the warning.

meethariprasad commented 4 years ago

We can close this issue if these warnings are expected and has no affect on fine tuning.

eduardofv commented 4 years ago

While the inexistent gradients problem does not affect training, it throws an error when reloading a keras model from a SavedModel file with custom object for the KerasLayer containing USE:

#This fails
model = tf.keras.models.load_model("model.h5",
                                   custom_objects={'KerasLayer':hub_layer})

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-f580b25d195c> in <module>()
      1 model = tf.keras.models.load_model("model.h5",
----> 2                                    custom_objects={'KerasLayer':hub_layer})

4 frames
/tensorflow-2.1.0/python3.6/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py in get_gradients(self, loss, params)
    395                            "gradient defined (i.e. are differentiable). "
    396                            "Common ops without gradient: "
--> 397                            "K.argmax, K.round, K.eval.".format(param))
    398       if hasattr(self, "clipnorm"):
    399         grads = [clip_ops.clip_by_norm(g, self.clipnorm) for g in grads]

ValueError: Variable <tf.Variable 'EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_0:0' shape=(31, 320) dtype=float32> has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

Loading with compile=False works, though. Is this also expected?

Check this Colab with complete example