exported huggingface tokenizer generates different results

[patricianing]

Certain fields in the tokenizer was not checked when exporting with onnxruntime-extension pnp module, causing a mismatch for cls_token and sep_token.

code showing the difference

import onnxruntime import onnxruntime_extensions import numpy as np

from onnxruntime_extensions import pnp from transformers import AutoTokenizer

def map_token_output(input_ids, attention_mask, token_type_ids): return input_ids.unsqueeze(0), token_type_ids.unsqueeze(0), attention_mask.unsqueeze(0)

model_name = 'sentence-transformers/all-mpnet-base-v2' output_name = 'all-mpnet-base-v2-aug.onnx'

symbolic = {0: 'batch_size', 1: 'sequence_length'}

tokenizer = AutoTokenizer.from_pretrained(model_name) bert_tokenizer = pnp.PreHuggingFaceBert(hf_tok=tokenizer)

augmented_model = pnp.SequentialProcessingModule(bert_tokenizer, map_token_output)

test_input = ["This is a test sentence"]

augmented_model = pnp.export(augmented_model, test_input, opset_version=12, input_names=['input'], output_names=['input_ids', 'attention_mask', 'token_type_ids'], output_path=output_name, dynamic_axes={'input_ids': symbolic, 'attention_mask': symbolic, 'token_type_ids': symbolic})

session_options = onnxruntime.SessionOptions() session_options.register_custom_ops_library(onnxruntime_extensions.get_library_path()) session = onnxruntime.InferenceSession(output_name, session_options) results = session.run([], {"input": test_input})

encoded_input = tokenizer(test_input, padding=True, truncation=True, return_tensors='pt') np.testing.assert_allclose(encoded_input.get('input_ids'), results[0], rtol=1e-04, atol=1e-05)

[patricianing]

Also tried passing the vocab file, no difference onnx_tokenizer = pnp.PreHuggingFaceBert(vocab_file='./all-mpnet-base-v2/vocab.txt')

[Craigacp]

I think this could be fixed if the tokenizer constructor was modified here to pull out the following variables:

            self.onnx_bert_tokenizer = create_op_function('BertTokenizer', bert_tokenizer,
                                                          hf_tok=hf_tok,
                                                          sep_token=hf_tok.eos_token,
                                                          cls_token=hf_tok.bos_token,
                                                          pad_token=hf_tok.pad_token)

As it defaults all those fields here but in the case of all-mpnet-base-v2 those defaults are wrong.