Converting models for tensoflowjs (node)

[ierezell]

Environment info

• transformers version: 4.3.3
• Platform: Linux-5.11.2-zen1-1-zen-x86_64-with-glibc2.2.5
• Python version: 3.8.8 (also 3.9.2 but tensorflowjs is not available in 3.9.2)
• PyTorch version (GPU?): 1.7.1 (True)
• Tensorflow version (GPU?): 2.4.1 (False)
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

• tensorflow: @jplu

Information

Model I am using (Bert, XLNet ...): Any tensorflow ones

The problem arises when using:

• [X] the official example scripts: (give details below)
• [X] my own modified scripts: (give details below)

The tasks I am working on is:

• [X] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Load a tensforflow model (e.g : model = TFAutoModel.from_pretrained("distilbert-base-uncased"))
2. Convert it to H5 with model.save_pretrained(path)
3. Try to load it in tensorflowjs and get UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'model_config' of null

Expected behavior

Be able to load a model in the LayerModel format as it's the only one which allows finetuning.

More informations.

I know the question was already posted in https://github.com/huggingface/transformers/issues/4073

I need to do finetuning so onnx, graphModels & cie should be avoided.

Seems that the H5 model just need the config file which seems saved on the side with the custom HF script. I went to read some issues on tensorflowjs (exemple : this or that) and the problem is that the HF model contains only the weights and not the architecture. The goal would be to adapt the save_pretrained function to save the architecture as well. I guess it's complex because of the Saving the model to HDF5 format requires the model to be a Functional model or a Sequential model. error in described bellow.

Seems also that only H5 model can be converted to a trainable LayerModel.

I'm willing to work on a PR or to help as i'm working on a web stack (nodejs) and I need this.

I made a drawing of all models (that I'm aware of) to summarize loading converting :

Also tried :

Use the nodejs tf.node.loadSavedModel which return only a saved model which I cannot use as the base structure with something like :

const bert = await tf.loadLayersModel(`file://${this._bert_model_path}/model.json`)
this._clf = tf.sequential();
this._clf.add(bert); // Raise Error
this._clf.add(tf.layers.dense({ units: 768, useBias: true }));
etc...
this._clf.compile(....)
this._clf.train(...)

Look for other libraries to train models (libtorch : incomplete, onnx training: only in python etc..)

Should I also write an issue on tensorflowjs ?

Thanks in advance for you time and have a great day.

[jplu]

Hello!

Thanks for reporting this issue! Did you try to convert your H5 file to be able to use it with tensorflowjs_converter --input_format keras path/to/my_model.h5 path/to/tfjs_target_dir? You can also have a SavedModel version with:

1. model = TFAutoModel.from_pretrained("distilbert-base-uncased")
2. model.save_pretrained(path, saved_model=True)

H5 and SavedModel conversion process are nicely explained in https://www.tensorflow.org/js/tutorials/conversion/import_keras and https://www.tensorflow.org/js/tutorials/conversion/import_saved_model

[ierezell]

Hi @jplu thanks for the fast reply !

Yes I used the tensorflowjs_converter to convert the H5 huggingFace model to a TFJS layerModel (.json).

If not using a h5 but a savedModel I don't know how to then use it in tensorflowJS.... all my researches seems to indicate that a savedModel can only do inference but i would like to finetune it.

My goal is just to finetune Bert in nodejs (could be with other framework than tfjs if you know some (like pytorch node bindings or some fancy framework but I found nothing that allow training))

Attempts :

HuggingFace :

1. Load model with huggingFace : model = TFAutoModel.from_pretrained(model_name)
2. Save it (H5 format) : model.save_pretrained(path_saved)
3. convert it to tfjs.json Layer format :

   dispatch_keras_h5_to_tfjs_layers_model_conversion(
   h5_path='path_saved/tf_model.h5',
   output_dir=my_tfjs_output_dir
   )

   (dispatch_keras_h5_to_tfjs_layers_model_conversion is the function called when using tensorflowjs_convert, i'm calling it from python code)

4. try to load it in nodejs : const bert = await tf.loadLayersModel('file://${my_tfjs_output_dir}/model.json')
5. Get the error UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'model_config' of null

Keras or tfjs saving (to have the model_config)

1. Load model with huggingFace : model = TFAutoModel.from_pretrained(model_name)
2. Save it (H5 format) :

   tf.keras.models.save_model(
   model=model,
   filepath=path_saved,
   signatures=tf.function(model.call).get_concrete_function([
       tf.TensorSpec([1, 512], tf.int32, name="input_ids"),
       tf.TensorSpec([1, 512], tf.int32, name="attention_mask")
   ]),
   save_format='h5'
   )

   or tfjs.converters.save_keras_model( model, path_saved)

3. Get the error Saving the model to HDF5 format requires the model to be a Functional model or a Sequential model.

Other formats :

1. Load model with huggingFace : model = TFAutoModel.from_pretrained(model_name)
2. Save it (SavedModel format) : model.save_pretrained(path_saved)
3. try to load it in nodejs : const bert = await tf.node.loadSavedModel(path_saved, ['serve'], "serving_default")
4. Cannot train it or use it in another model with this._clf = tf.sequential(); this._clf.add(bert);

GraphModel or onnx model are for inference only

Sorry for this (too) long reply.

Thanks for your help (I guess many people try to attempt the same),

Have a great day

[jplu]

Indeed you cannot train or fine tune a SavedModel in other envs than Python.

Currently to load a model you are forced to have the config file because it is required as an argument to init a model. Then the H5 as well. Furthermore, the TF implementations are using subclass models which are not compliant with most of the internal TF process (such as what you are trying to do in your described process when saving a model).

Then, I suggest you to fine tune your model in Python, create a SavedModel, and then use it in JS as described in the link I shared. Sorry for the inconvenience.

[ierezell]

Okay so SOA is training custom models in tfjs but finetunning is almost impossible. It's what I thought but I'm glad you confirmed it.

Do you know any other solutions that would allow me to finetune a model in a webstack (nodejs / typescript) ? I'm still hoping that one day pytorch will have node bindings...

Thanks again for your time.
I guess we can close the issue.

[jplu]

Do you know any other solutions that would allow me to finetune a model in a webstack (nodejs / typescript) ?

No, sorry this is not really my domain. And sorry again for the inconvenience on this.