search

keras-team / keras-hub

Pretrained model hub for Keras 3
Apache License 2.0
790 stars 242 forks source link

TransformerEncoder layer unable to be saved #1303

Open mateuszlugowski opened 1 year ago

mateuszlugowski commented 1 year ago

Describe the bug

Trying to save a model containing TransformerEncoder layer results in an error. This is due to some weights of it having the same exact name, and h5 format is raising a ValueError because of that.

To Reproduce

from keras_nlp.layers import TransformerEncoder

inp = keras.layers.Input((30, 128))
tl = TransformerEncoder(
                intermediate_dim=512,
                num_heads=8,
            )
model = keras.models.Model(inp, tl(inp))
model.save_weights('test.h5')
pranavvp16 commented 1 year ago

I also encountered the same error while implementing ELECTRA in keras. The previous version of TransformerEncoder was able to save successfully which I found from this notebook written by @abheesht17 , which implements previous version of keras-nlp. I tried to run the same notebook with the latest version of keras-nlp and saving of TransformerEncoder in .h5 failed.
The only change I made in the new notebook was rename _self_attention_layernorm to _self_attention_layer_norm which is the according to the latest naming in TransformerEncoder. notebook2