I don't believe this approach correctly implements transfer learning/fine-tuning. I believe you are meant to freeze/fix the convolutional layers and train only the logits layer. You can update the code by changing the calls to compute_gradients as such:
rgb_grads = opt_rgb.compute_gradients(rgb_loss)
rgb_grads = opt_rgb.compute_gradients(rgb_loss, var_list=[v for v in tf.trainable_variables() if "Logits" in v.name])
I don't believe this approach correctly implements transfer learning/fine-tuning. I believe you are meant to freeze/fix the convolutional layers and train only the logits layer. You can update the code by changing the calls to compute_gradients as such: