I have my own pretrained Pegasus model, now I want to finetune using BigBird, so this is my mapping function,
import re
import collections
def get_assignment_map_from_checkpoint(tvars, init_checkpoint):
"""Compute the union of the current variables and checkpoint variables."""
assignment_map = {}
initialized_variable_names = {}
name_to_variable = collections.OrderedDict()
for var in tvars:
name = var.name
m = re.match('^(.*):\\d+$', name)
if m is not None:
name = m.group(1)
name_to_variable[name] = var
init_vars = tf.train.list_variables(init_checkpoint)
assignment_map = collections.OrderedDict()
for x in init_vars:
(name, var) = (x[0], x[1])
l = 'pegasus/' + name
l = l.replace('embeddings/weights', 'embeddings/word_embeddings')
l = l.replace('self/output', 'output')
l = l.replace('ffn/dense_1', 'output/dense')
l = l.replace('ffn', 'intermediate')
l = l.replace('memory_attention/output', 'attention/encdec_output')
l = l.replace('memory_attention', 'attention/encdec')
if l not in name_to_variable:
continue
assignment_map[name] = name_to_variable[l]
initialized_variable_names[l + ':0'] = 1
return (assignment_map, initialized_variable_names)
I have my own pretrained Pegasus model, now I want to finetune using BigBird, so this is my mapping function,
output,
My pegasus config, Copy pasted from https://github.com/google-research/bigbird/blob/master/bigbird/summarization/pegasus_large.sh
Not sure this is the correct one, finetuning is really slow, so any guide about variable mapping is really helpful.