How to export a Huggingface transfromer model?

[vukovinski]

Tutorial Select

Prepare Custom Model

Feedback

I would like more context on how to optimize for mobile models other than the ones registered with PyTorch hub, such as Hugging Face Hub for example.

[vukovinski]

I tried something like the code below, but would get unknown tensor error. Could be a numpy interaction.

import torch
import numpy as np
from transformers import AutoModelForSequenceClassification
from torch.utils.mobile_optimizer import optimize_for_mobile

task='sentiment'
MODEL = f"cardiffnlp/twitter-roberta-base-{task}"
model = AutoModelForSequenceClassification.from_pretrained(MODEL)
model.eval()

scripted_model = torch.jit.script(model)
optimized_model = optimize_for_mobile(scripted_model)
optimized_model._save_for_lite_interpreter("roberta-base-sentiment.ptl")

Traceback (most recent call last):
  File "C:\Users\filip\Desktop\Ignition\roberta_exporter.py", line 19, in <module>
    scripted_model = torch.jit.script(model)
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_script.py", line 1284, in script
    return torch.jit._recursive.create_script_module(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 480, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 542, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_script.py", line 614, in _construct
    init_fn(script_module)
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 520, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 542, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_script.py", line 614, in _construct
    init_fn(script_module)
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 520, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 546, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\jit\_recursive.py", line 397, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError: 
Expected a default value of type Tensor (inferred) on parameter "past_key_values_length".Because "past_key_values_length" was not annotated with an explicit type it is assumed to be type 'Tensor'.:       
  File "C:\Users\filip\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\models\roberta\modeling_roberta.py", line 95
    def forward(
    ~~~~~~~~~~~~
        self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, past_key_values_length=0
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ):
    ~~
        if position_ids is None:
        ~~~~~~~~~~~~~~~~~~~~~~~~
            if input_ids is not None:
            ~~~~~~~~~~~~~~~~~~~~~~~~~
                # Create the position ids from the input token ids. Any padded tokens remain padded.
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            else:
            ~~~~~
                position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        if input_ids is not None:
        ~~~~~~~~~~~~~~~~~~~~~~~~~
            input_shape = input_ids.size()
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        else:
        ~~~~~
            input_shape = inputs_embeds.size()[:-1]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        seq_length = input_shape[1]
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~

        # Setting the token_type_ids to the registered buffer in constructor where it is all zeros, which usually occurs
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        # issue #5664
        ~~~~~~~~~~~~~
        if token_type_ids is None:
        ~~~~~~~~~~~~~~~~~~~~~~~~~~
            if hasattr(self, "token_type_ids"):
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                buffered_token_type_ids = self.token_type_ids[:, :seq_length]
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                buffered_token_type_ids_expanded = buffered_token_type_ids.expand(input_shape[0], seq_length)
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                token_type_ids = buffered_token_type_ids_expanded
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            else:
            ~~~~~
                token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        if inputs_embeds is None:
        ~~~~~~~~~~~~~~~~~~~~~~~~~
            inputs_embeds = self.word_embeddings(input_ids)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        token_type_embeddings = self.token_type_embeddings(token_type_ids)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        embeddings = inputs_embeds + token_type_embeddings
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        if self.position_embedding_type == "absolute":
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            position_embeddings = self.position_embeddings(position_ids)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            embeddings += position_embeddings
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        embeddings = self.LayerNorm(embeddings)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        embeddings = self.dropout(embeddings)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        return embeddings
        ~~~~~~~~~~~~~~~~~ <--- HERE

[rraihansaputra]

Hi @vukovinski , this process would be different from model to model. What I can describe from a high level perspective is that the torch.jit.script tries to convert the PyTorch model into a TorchScript model. The key process is inferring the types of variables in the model process, and in almost every case you would need to modify the model definition to add the type annotations so the .script function can correctly determine the types of the variables.

An easier way is to use the torch.jit.trace function but it also has their own limitations listed in the documentation.

[vukovinski]

I see, I believe I understand now. Basically I would need access to the source code for the model so that I could annotate it with python or torchscript (some confusion here) datatypes. Hmm, this requirement precludes using HuggingFace models for mobile device optimization, unless the original authors of the models embrace such a practice.

Alternatively, such models (meaning weights and nodes) could be reverse engineered, but that sounds like more trouble than retraining them.

Thank you once again @rraihansaputra

Move to close the issue.