google-deepmind / kinetics-i3d

Convolutional neural network model for video classification trained on the Kinetics dataset.
Apache License 2.0
1.75k stars 462 forks source link

code to load 2D parameters (pretrained on imagenet) to 3D model #52

Open emergencyd opened 5 years ago

emergencyd commented 5 years ago

Could you please provide the code which loads 2D parameters (pretrained on imagenet) to I3D model? (especially the processing of BN/GN).

Thank you very much!

joaoluiscarreira commented 5 years ago

Hi,

this is old tensorflow code (2017), but hopefully still helpful:

def assign_from_checkpoint_2d_to_3d_scale(model_path, var_list): """Creates an operation to assign specific variables from a checkpoint.

Args: model_path: The full path to the model checkpoint. To get latest checkpoint use model_path = tf.train.latest_checkpoint(checkpoint_dir) var_list: A list of Variable objects or a dictionary mapping names in the checkpoint to the corresponding variables to initialize. If empty or None, it would return no_op(), None.

Returns: the restore_op and the feed_dict that need to be run to restore var_list.

Raises: ValueError: If the checkpoint specified at model_path is missing one of the variables in var_list. """

reader = pywrap_tensorflow.NewCheckpointReader(model_path)

if isinstance(var_list, (tuple, list)): var_list = {var.op.name: var for var in var_list}

feed_dict = {} assign_ops = []

for checkpoint_var_name in var_list: var = var_list[checkpoint_var_name] if not reader.has_tensor(checkpoint_var_name): raise ValueError( 'Checkpoint is missing variable [%s]' % checkpoint_var_name)

var_value = reader.get_tensor(checkpoint_var_name)
placeholder_name = 'placeholder/' + var.op.name
placeholder_value = tf.placeholder(
    dtype=var.dtype.base_dtype,
    shape=var.get_shape(),
    name=placeholder_name)
assign_ops.append(var.assign(placeholder_value))

if var.get_shape() != var_value.shape:
  n = var_value.shape[0]
  feed_dict[placeholder_value] = np.tile(np.expand_dims(var_value/n,0), [n,1,1,1,1])
else:
  feed_dict[placeholder_value] = var_value.reshape(var.get_shape())

assign_op = tf.group(*assign_ops) return assign_op, feed_dict

emergencyd commented 5 years ago

Thank you very much! Is there bn/gn parameter in the pretrained 2D model? if so, do we need to ignore it or also load it to the I3D model?

joaoluiscarreira commented 5 years ago

There was batch norm in the pretrained 2d model, and i think that code loads the corresponding parameters. We also didn't freeze the batch norm parameters when finetuning on any of the datasets in the paper.

Joao

On Wed, Feb 20, 2019 at 2:04 PM emergencyd notifications@github.com wrote:

Thank you very much! Is there bn/gn parameter in the pretrained 2D model? if so, do we need to ignore it or also load it to the I3D model?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/deepmind/kinetics-i3d/issues/52#issuecomment-465587915, or mute the thread https://github.com/notifications/unsubscribe-auth/AO6qau9nCYsIqeG2-ypPeT6T-u1IXTcLks5vPVYJgaJpZM4bFJV5 .