model.eval(): AttributeError: 'function' object has no attribute 'eval'

[Matthieu-Tinycoaching]

Libraries versions:
neuron-cc = 1.3.7.0
torch-neuron == 1.7.1.1.3.5.0

Hello,

I compiled a sentence-transformer model with neuroncore_pipeline_cores = 4 and batch_size = 6 according the code below:

import tensorflow  # to workaround a protobuf version conflict issue
import torch
import torch.neuron
import transformers

#### I/ Compile the sts-transformer with torchscript ####

model_name = "sentence-transformers/stsb-xlm-r-multilingual"
pipeline_name = "feature-extraction"
nlp = transformers.pipeline(pipeline_name, model=model_name, tokenizer=model_name, device=-1)
tokenizer = nlp.tokenizer

class SentenceTransformerCustom(transformers.XLMRobertaModel):
    def __init__(self, config):
        super().__init__(config)
        # Naming alias for ONNX output specification
        # Makes it easier to identify the layer
        self.sentence_embedding = torch.nn.Identity()

    def forward(self, input_ids, attention_mask):
        # Get the token embeddings from the base model
        token_embeddings = super().forward(
            input_ids, 
            attention_mask=attention_mask
        )[0]
        # Stack the pooling layer on top of it
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
        sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
        return self.sentence_embedding(sum_embeddings / sum_mask)

model = SentenceTransformerCustom(config=nlp.model.config).from_pretrained(model_name, return_dict=False)

src_text = "Adresse électronique : Pour pouvoir s'échanger des messages électroniques il faut une adresse... électronique ! Tout comme avoir une adresse postale est mieux pour recevoir une lettre. L'adresse électronique peut être obtenue auprès d'un fournisseur de messagerie (Gmail, Hotmail, etc.), et se découpe en 3 parties : l'identifiant que vous avez choisi (ex. : bobforever), l'arobase (@), et le nom de domaine (ex. : gmail.com)."

max_length=128
dummy_phrase = tokenizer(src_text, padding='max_length', truncation=True, max_length=max_length, return_tensors='pt')

batch_size = 6
example_inputs_dummy_phrase = (
    torch.cat([dummy_phrase['input_ids']] * batch_size,0),
    torch.cat([dummy_phrase['attention_mask']] * batch_size,0)
)

neuroncore_pipeline_cores = 4 # This string should be '4' on an inf1.xlarge
neuron_pipeline_model = torch.neuron.trace(model,
                                        example_inputs = example_inputs_dummy_phrase,
                                        verbose=1,
                                        compiler_args = ['--neuroncore-pipeline-cores', str(neuroncore_pipeline_cores)],
                                        compiler_workdir='./'
                                        )
neuron_pipeline_model.save("stsb-xlm-r-multilingual-custom-neuron-pipeline{}-parallel-pad128-b{}.pt".format(neuroncore_pipeline_cores, batch_size))

Compilation succeeded but once I load the model and try to put it in evaluation mode:

model = torch.jit.load
('stsb-xlm-r-multilingual-custom-neuron-pipeline4-parallel-pad128-b6.pt')
model.eval()

I got the following error:

    model.eval()
AttributeError: 'function' object has no attribute 'eval'

I sent to you the model file at aws-neuron-support@amazon.com in order for you to debug.

Thanks in advance!

[jeffhataws]

Hi Matthieu,

We were able to take you code and the stsb-xlm-r-multilingual-custom-neuron-pipeline4-parallel-pad128-b6-local.pt example you provided via email and run inference with this code:

import torch
import torch_neuron
import transformers

# Create tokenizer and example inputs
model_name = "sentence-transformers/stsb-xlm-r-multilingual"
pipeline_name = "feature-extraction"
nlp = transformers.pipeline(pipeline_name, model=model_name, tokenizer=model_name, device=-1)
tokenizer = nlp.tokenizer

src_text = "Adresse électronique : Pour pouvoir s'échanger des messages électroniques il faut une adresse... électronique ! Tout comme avoir une adresse postale est mieux pour recevoir une lettre. L'adresse électronique peut être obtenue auprès d'un fournisseur de messagerie (Gmail, Hotmail, etc.), et se découpe en 3 parties : l'identifiant que vous avez choisi (ex. : bobforever), l'arobase (@), et le nom de domaine (ex. : gmail.com)."

max_length=128
dummy_phrase = tokenizer(src_text, padding='max_length', truncation=True, max_length=max_length, return_tensors='pt')

batch_size = 6

example_inputs_dummy_phrase = (
    torch.cat([dummy_phrase['input_ids']] * batch_size,0),
    torch.cat([dummy_phrase['attention_mask']] * batch_size,0)
)

# Load the model
model = torch.jit.load('stsb-xlm-r-multilingual-custom-neuron-pipeline4-parallel-pad128-b6-local.pt')
print('Successfully loaded the model')

# Set to eval mode
model.eval()
print('Successfully set the model to `.eval()` mode')

output = model(*example_inputs_dummy_phrase)
print('Inference result is:\n', output)
print('Successfully ran inference!')

The torch-neuron version tested is like what you reported:

(customer_env) ubuntu@ip-172-31-63-138:~$ pip list | grep torch
torch                1.7.1
torch-neuron         1.7.1.1.3.5.0
torchvision          0.8.2

Could you try running this code and see if your problem is resolved?

Thanks.

[Matthieu-Tinycoaching]

Hi @jeffhataws thanks it works for me too.

I had a bug of syntax in my code line :

model = torch.jit.load
('stsb-xlm-r-multilingual-custom-neuron-pipeline4-parallel-pad128-b6-local.pt')

instead of: model = torch.jit.load('stsb-xlm-r-multilingual-custom-neuron-pipeline4-parallel-pad128-b6-local.pt')