keras-team / keras-io

Keras documentation, hosted live at keras.io
Apache License 2.0
2.69k stars 2.01k forks source link

modify mlm example to support all backends #1856

Open Mrutyunjay01 opened 1 month ago

Mrutyunjay01 commented 1 month ago

With reference to the draft PR, here's an attempt to make to example truly backend-agnostic. In the process, I raised the following issues: https://github.com/keras-team/keras/issues/18410, and https://github.com/keras-team/keras/issues/19665. While the latter got resolved, the former persists still. Kindly review the changes, and let me know how shall we proceed with the issues mentioned.

cc: @fchollet

Mrutyunjay01 commented 1 month ago

added subclassing implementation. Certain obstacles remain as mentioned:

  1. In BertEncoderLayer class, as it is inherited from layers.Layer, is it possible to access to the sub-layers here? as:
    
    mlm_model = models.load_model("bert_mlm.keras", custom_objects={"MaskedLanguageModel": MaskedLanguageModel})

mlm_model.layers :

[,

,

,

]

mlm_model.get_layer("bert_encoder_layer").get_layer("encoder_0_ffn_layernormalization")


This throws error saying 

```py
AttributeError: 'BertEncoderLayer' object has no attribute 'get_layer'
  1. The reason I want to do the above is to reproduce the following:
    # Load pretrained bert model
    mlm_model = models.load_model(
    "bert_mlm_imdb.keras", custom_objects={"MaskedLanguageBertModel": MaskedLanguageBertModel}
    )
    pretrained_bert_model = Model(
    mlm_model.input, mlm_model.get_layer("encoder_0_ffn_layernormalization").output
    )

This was possible in functional declaration of the Model, but not in sub-classing (in last commit).

Kindly suggest w.r.t last commit

Thanks!