Open pjlintw opened 1 year ago
Same problem encountered. Have you solved it? @pjlintw
Same issue. Did anyone resolve? It seems that the variables
ckpt_state_dict
and dummy_written_state_dict
have different structures in the file t5x/checkpoints.py
(line 1119) :
ckpt_state_dict
:
state
param_states
decoder
decoder_norm
scale
m
(1,)
v
(768,)
v_col
(1,)
v_row
(1,)
layers_0
encoder_decoder_attention
key
kernel
m
(1,)
v
(1,)
v_col
(768,)
v_row
(768,)
out
kernel
m
(1,)
.
.
.
.
encoder
encoder_norm
scale
m
(1,)
v
(768,)
v_col
(1,)
v_row
(1,)
layers_0
attention
key
kernel
m
(1,)
v
(1,)
v_col
(768,)
v_row
(768,)
out
kernel
m
(1,)
v
(1,)
v_col
(768,)
v_row
(768,)
target
decoder
decoder_norm
scale
(768,)
layers_0
encoder_decoder_attention
key
kernel
driver
And for dummy_written_state_dict
:
state
param_states
1
0
count
mu
decoder
decoder_norm
scale
layers_0
encoder_decoder_attention
key
kernel
out
kernel
.
.
.
.
nu
decoder
decoder_norm
scale
layers_0
encoder_decoder_attention
key
kernel
out
kernel
query
kernel
value
kernel
mlp
wi
kernel
wo
kernel
pre_cross_attention_layer_norm
scale
pre_mlp_layer_norm
scale
pre_self_attention_layer_norm
scale
self_attention
key
kernel
target
decoder
decoder_norm
scale
layers_0
encoder_decoder_attention
key
kernel
out
kernel
query
kernel
.
.
.
.
@pjlintw @kiriharulxh
trying to fine-tune T5-small v1.1 on single GPU using on the sample script (singlenode_ft_frompile.sh)
Not completely know how the script loads the checkpoint but I've set the
GOOGLE_CLOUD_BUCKET_NAME
and it seems the script could find the checkpoint (gs://t5-data/pretrained_models/t5x/t5_1_1_small
) on the google cloud storage.It appears that there is an issue where the target dictionary cannot load the weights from the state dictionary, possibly due to an additional key '{'1'}' in the target dictionary.
I get the following error:
and set-up for the command:
Also, still got the same error as trying the base size of the model.