Using with a custom model not on HuggingFace

[coogle]

Your question

This is an awesome project, kuddos to you. I hope you can help me use it!

I've created a custom model for multi text classification that I've exported as an .onnx file (originally made in sklearn). I've tried to import this file a bunch of different ways to use it and I've thus far been unsuccessful.

Can you provide some guidance about how one might use a custom model? I can't find much useful documentation on how to put together config.json and such needed here... plus, it seems like most of the documentation right now is geared toward models published on HuggingFace -- rather than proprietary custom models.

End goal here is I have a bunch of training data that takes NLP and assigns it 0 or more custom (known) tags. I'd like to be able to use this model in a PHP request to create tags from input text.

Context (optional)

model = clr.fit(
    cv_clean.transform(X_train_clean),
    y_train
)

initial_type = [('float_input', FloatTensorType([None, cv_clean.transform(X_train_clean).shape[1]]))]

onnx_model = convert_sklearn(model, initial_types=initial_type, target_opset=20, options={type(clr.estimator): {'zipmap': False}})

onnx_model_path = "model_quantized.onnx"
with open(onnx_model_path, "wb") as f:
    f.write(onnx_model.SerializeToString())

Reference (optional)

No response

[coogle]

To be a little more clear my thoughts here are:

1. Create a model (in this case I used sklearn and then converted it to a .onnx)
2. Deploy the model in my PHP app by storing it where transformers-php expects to find it (in my case, storage/app/transformers/coogle/tags-classification

Ultimately I'm trying to create an interface such that:

$classifier = pipeline('text-classification', $model);
$result = $classifier($input);

Where $result will be the tags the model predicted should be applied to $input.

I suspect looking into the code I need to manually use some sub-classes and not rely on the pipeline helper.

FWIW this is literally my very first attempt at training my own model, so it's entirely possible I have no idea what I'm doing here and any pointers in the right direction to getting this specific use case working would be extremely appreciated.

Once I get my head around how this all works I'm very happy to help improve this code base with some PRs to make it easier for the next person that comes by.

[CodeWithKyrian]

Hi @coogle ,

Thanks for your interest in TransformersPHP! It's great to see you exploring custom model deployment. You're right - our docs mostly focus on HuggingFace models since that's the common use case. There's a little part that discussed using custom models though, but I must admit that it's not as detailed as it needs to be.

The pipeline function may not be the best thing to use in this case. It's more of a helper method that wraps the complex processes involved in inferencing from preprocessing inputs to post processing outputs. For text classification however, the process is fairly simple - tokenization, inference proper, and post-processing. For custom usage, it's recommended to use the individual model classes.

Tokenizer

First, you’ll need to load the tokenizer with AutoTokenizer. Ensure you have a tokenizer.json in your model folder, so the class picks it up locally instead of trying to reach the Hub. The structure of this file depends on the tokenization process you used when training.

Here’s a breakdown of the top-level keys in tokenizer.json:

1. Normalizer: This step involves normalizing the input text (e.g., lowercasing, Unicode normalization). Example: BertNormalizer.
2. Pre-Tokenizer: This step splits the input text into tokens. Example: Whitespace.
3. Model: This is the core of the tokenizer, mapping tokens to IDs. The model type can be Unigram, WordPiece, or BPE (Byte-Pair Encoding). Example: WordPiece.
4. Post-Processor: This step modifies the tokens after initial tokenization. Example: TemplateProcessing.
5. Decoder: This step converts token IDs back to text. Example: BPEDecoder, ByteFallback, FuseDecoder, MetaspaceDecoder

TransformersPHP supports these types, and you should match the values in your tokenizer.json accordingly. Check out the toxic-bert tokenizer for an example. In my experience, the Model is the most important, for the rest, you can make them null

Here’s an example structure for a tokenizer.json file:

{
  "normalizer": {
    "type": "BertNormalizer",
    "clean_text": true,
    "handle_chinese_chars": true,
    "strip_accents": null,
    "lowercase": true
  },
  "pre_tokenizer": {
    "type": "Whitespace"
  },
// "pre_tokenizer":  null (it could be null too)
  "model": {
    "type": "WordPiece",
    "vocab": {
      "[PAD]": 0,
      "[UNK]": 1,
      "[CLS]": 2,
      "[SEP]": 3,
      "[MASK]": 4,
      "the": 5,
      "to": 6,
      // additional vocabulary entries
    },
    "unk_token": "[UNK]",
    "max_input_chars_per_word": 100
  },
  "post_processor": {
    "type": "TemplateProcessing",
    "single": "[CLS] $A [SEP]",
    "pair": "[CLS] $A [SEP] $B:1 [SEP]:1",
    "special_tokens": {
      "[CLS]": 2,
      "[SEP]": 3
    }
  },
  "decoder": {
    "type": "WordPiece",
    "cleanup": true
  }
}

If everything is setup right, then you can create the tokenizer like so:

$tokenizer = AutoTokenizer::fromPretrained('coogle/tags-classification');

PS: Could you please explain your tokenization process? This way, I can list the supported types for each stage that match your method.

Model

Next, load your model with AutoModel::fromPretrained. You'll need a config.json in your model directory, or you can just pass an array with necessary configs and it'll skip looking for the JSON file. If you check example config.json like this, there are a lot of keys as well. Most of them are relevant for different tasks. In this case (text-classification), the most relevant keys are the label2id and id2label (which map string labels to numerical ids that the model will output and vice versa respectively).

$model = AutoModel::fromPretrained(
    'coogle/tags-classification',
    config: [
        'label2id' => ['tag1' => 0, 'tag2' => 1, 'tag3' => 2],
        'id2label' => ['0' => 'tag1', '1' => 'tag2', '2' => 'tag3']
    ],
    modelFilename: 'model_quantized.onnx'
);

Inference

Tokenize your input text and perform inference:

$modelInputs = $tokenizer->tokenize($textInputs, padding: true, truncation: true);
$outputs = $model($modelInputs);
// The proceed compute softmax of the output logits to get the predictions, then map the ids to corresponding labels.

Using Pipeline with Custom Models

Although not in the documentation, you can actually manually construct the pipeline (without the pipeline helper method this time around). You'll use the model and tokenizer instance this time around.

$classifier = new TextClassificationPipeline('text-classification', $model, $tokenizer);
$result = $classifier($input);

This lets you leverage the pipeline's functionality while using your custom model and tokenizer.

Feel free to reach out if you have more questions.

[coogle]

Thanks for getting back to me! I really appreciate this help it's exactly what I'm missing.

When I tried to do as you suggested I'm getting an error:

($labels is my labels as an id => string array -- my model_quantized.onnx is in coogle/tags-classifier/onnx/model_quantized.onnx)

            $modelObj = AutoModel::fromPretrained(
                $model,
                config: [
                    'id2label' => $labels,
                    'label2id' => array_flip($labels),
                ],
            );

 TypeError  Cannot assign null to property Codewithkyrian\Transformers\Utils\AutoConfig::$modelType of type string.

--
 () at vendor/codewithkyrian/transformers/src/Utils/AutoConfig.php:27

It seems that AutoConfig requires modelType to not be null here -- I tried looking into this code but I didn't know enough about what exactly you were doing here to understand if modelType should be able to be null and it's a typing issue, or if I was doing something wrong here.

Regarding the tokenizer, I'm using:

from sklearn.feature_extraction.text import TfidfVectorizer # Tokenizes the data based on a term-frequency-inverse-document algorithm

I haven't figured out how to generate a tokenizer.json from this yet (suggestions most welcome)

[coogle]

Just to give you some more insight here's the python script I'm using right now... pardon the horrible code as I hack around (you can see where I was playing around with generating a tokenizer.json which is obviously wrong)

#!/usr/bin/env python
# coding: utf-8

# In[1]:

import warnings
import os
import pandas as pd # For loading data
import nltk # NLTK stands for Natural Language Took Kit
from nltk.corpus import stopwords
import re # Regex
import string # String library has more string manipulation features
import numpy as np # Numerical Python library
import matplotlib.pyplot as plt # For graphing
import seaborn as sns # For graphing
from sklearn.feature_extraction.text import TfidfVectorizer # Tokenizes the data based on a term-frequency-inverse-document algorithm
from sklearn.feature_extraction.text import CountVectorizer # Implements tokenization and counts all in one
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.metrics import multilabel_confusion_matrix
from sklearn.multioutput import MultiOutputClassifier # Multi-output wrapper import
from sklearn.naive_bayes import MultinomialNB # Naive Bayes import
from sklearn.svm import LinearSVC # Support Vector Machine import
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score # Metrics
from sklearn.linear_model import LogisticRegression # Logistic regression import
from skl2onnx import to_onnx
import onnxruntime as rt
import onnx
import skl2onnx
from skl2onnx.common.data_types import FloatTensorType
from skl2onnx import to_onnx
from skl2onnx import convert_sklearn
import json
from transformers import AutoTokenizer

# In[2]:

warnings.filterwarnings("ignore")

# In[3]:

nltk.download('stopwords')

# In[4]:

df = pd.read_csv('model-data/tags-training-data.csv')
df = df.convert_dtypes()
df

# In[5]:

df_tags_trans = df.iloc[:, 1:].T.sum(axis=1).sort_values(ascending=False).copy()
df_tags_trans

# In[6]:

stop_words = stopwords.words('english')

def clean_text(text):
    text = str(text).lower() # Lowercases everything 
    text = re.sub('<.*?>+', '', text) # Removes everything between '<' and '>'
    text = re.sub(r'[^\w\s]', '', text) # Removes all punctuation
    text = re.sub('\n', '', text) # Removes any breaklines
    text = re.sub('\w*\d\w*', '', text) # Removes any words that contain numbers 

    # Remove Stopwords
    text = ' '.join(word for word in text.split() if word not in stop_words)

    # Removing non english characters
    text = ''.join(line for line in text if line in string.printable)

    # Fixing extra whitespaces between words
    text = ' '.join(text.split())
    return text

# In[7]:

df['query_cleaned'] = df['query'].apply(clean_text)
df[['query', 'query_cleaned']]

# In[8]:

df = df.sample(df.shape[0], random_state=43)
df = df.reset_index(drop=True)
df

# In[9]:

percent = 0.9
num_training_examples = int(df.shape[0] * percent)
df_train = df.iloc[:num_training_examples]
df_test = df.iloc[num_training_examples:]

# In[10]:

df_train.shape

# In[11]:

df_test.shape

# In[12]:

X_raw = df['query']
X_clean = df['query_cleaned']
y = df.columns[1:]

X_train_raw = df_train['query']
X_train_clean = df_train['query_cleaned']

X_test_raw = df_test['query']
X_test_clean = df_test['query_cleaned']

y_train = df_train.iloc[:, 1:-1].astype(int)
y_test = df_test.iloc[:, 1:-1].astype(int)

# In[32]:

tfidf_raw = TfidfVectorizer()
tfidf_raw.fit_transform(X_train_raw)

tfidf_clean = TfidfVectorizer()
tfidf_clean.fit_transform(X_train_clean)

cv_raw = CountVectorizer()
cv_raw.fit_transform(X_train_raw)

cv_clean = CountVectorizer()
cv_clean.fit_transform(X_train_clean)

# In[29]:

def smaller_classification_report(y_true, y_pred, floating_point = 2):
    """
    Function to help us print a nicer classification if we just want selected metrics
    """
    print("Accuracy \t\t{}".format(round(accuracy_score(y_true, y_pred), floating_point)))
    print("Precision (weighted) \t{}".format(round(precision_score(y_true, y_pred, average='weighted'), floating_point)))
    print("Recall (weighted) \t{}".format(round(recall_score(y_true, y_pred, average='weighted'), floating_point)))
    print("F1-score (weighted) \t{}".format(round(f1_score(y_true, y_pred, average='weighted'), floating_point)))

# In[33]:

# Creating the chain model
clr = MultiOutputClassifier(
    LogisticRegression(max_iter=500)
)
model = clr.fit(
    #cv_clean.transform(X_train_clean), # Uncomment to try and comment out below
    tfidf_clean.transform(X_train_raw), 
    y_train
)
# Making predictions
y_pred = model.predict(
    #cv_clean.transform(X_test_clean), # Uncomment to try and comment out below
    tfidf_clean.transform(X_test_raw)
)

smaller_classification_report(y_test.values.astype(int), y_pred)

# In[34]:

# Define the initial types for the ONNX model
initial_type = [('float_input', FloatTensorType([None, tfidf_clean.transform(X_train_clean).shape[1]]))]

# Convert the trained model to ONNX format
onnx_model = convert_sklearn(model, initial_types=initial_type, target_opset=20, options={type(clr.estimator): {'zipmap': False}})

# Save the ONNX model to a file
onnx_model_path = "build/Arvee/tags-classifier/onnx/model_quantized.onnx"
with open(onnx_model_path, "wb") as f:
    f.write(onnx_model.SerializeToString())

print(f"ONNX model saved to {onnx_model_path}")

labels = list(y_train)
id2label = {i: label for i, label in enumerate(labels)}
label2id = {label: i for i, label in enumerate(labels)}

# Create the configuration files for transformers library
config = {
    "model_type": "text-classification",
    "vocab_size": len(tfidf_clean.vocabulary_),
    "hidden_size": tfidf_clean.transform(X_train_clean).shape[1],
    "id2label": id2label,
    "label2id": label2id
}

# Save the configuration to a JSON file
config_path = "build/Arvee/tags-classifier/config.json"
with open(config_path, "w") as f:
    json.dump(config, f, indent=2)

print(f"Configuration saved to {config_path}")

tokenizer_path = "build/Arvee/tags-classifier/"
# Load a predefined tokenizer and save its files
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Save the tokenizer to the current directory
tokenizer.save_pretrained(tokenizer_path)

print(f"Tokenizer Metadata saved to {tokenizer_path}")

# Load the ONNX model using ONNX Runtime
session = rt.InferenceSession(onnx_model_path)

# Prepare the test data for ONNX Runtime
X_test_onnx = tfidf_clean.transform(X_test_clean).toarray().astype(np.float32)

# Execute the model against the test data
input_name = session.get_inputs()[0].name
label_name = session.get_outputs()[0].name

y_pred_onnx = session.run([label_name], {input_name: X_test_onnx})[0]

# Print the classification report
smaller_classification_report(y_test.values.astype(int), y_pred_onnx)

[CodeWithKyrian]

Great info!

First off, you could fix the error by setting model_type as custom in the config array you passed to the AutoModel. But then, in your training script, I noticed you were already saving a config.json, so you needn't pass the config array to the AutoModel anymore. Just make sure the saved config.json is in the right directory and it'll load the config from there.

labels = list(y_train.columns)
id2label = {i: label for i, label in enumerate(labels)}
label2id = {label: i for i, label in enumerate(labels)}

config = {
    "model_type": "custom",
    "vocab_size": len(tfidf_clean.vocabulary_),
    "hidden_size": tfidf_clean.transform(X_train_clean).shape[1],
    "id2label": id2label,
    "label2id": label2id
}

config_path = "build/Arvee/tags-classifier/config.json"
with open(config_path, "w") as f:
    json.dump(config, f, indent=2)

Then for the tokenizer, I'd suggest constructing one from scratch for now, instead of reusing the BertTokenizer. I'll try and walk you through the different parts of it, but do some research to back it up later as well.

Normalizer

Your clean_text function performs several normalization steps like lowercasing, removing HTML tags, punctuation, line breaks, and stopwords. This can be represented using a combination of normalizers. For eg.

"normalizer": {
  "type": "Sequence",
  "normalizers": [
    {"type": "NFD"},
    {"type": "Lowercase"},
    {"type": "StripAccents"}
  ]
}

But there's a predefined normalizer, BertNormalizer that can handle lowercasing, stripping accents, and handling Chinese characters so I'd go for that instead.

"normalizer": {
    "type": "BertNormalizer",
    "clean_text": true,
    "handle_chinese_chars": true,
    "strip_accents": true,
    "lowercase": true
  },

For the stop words and removing HTML Characters, you can use a couple of Replace normalizers. The Replace normalizer expects a pattern with a String or Regex key. Eg.

{
  "normalizer": {
    "type": "Sequence",
    "normalizers": [
      {
        "type": "Replace",
        "pattern": {
          "Regex": "<.*?>"
        },
        "content": ""
      },
      {
        "type": "Replace",
        "pattern": {
          "Regex": "\\b(" + "|".join(stop_words) + ")\\b"
        },
        "content": ""
      },
      {
        "type": "BertNormalizer",
        "clean_text": true,
        "handle_chinese_chars": true,
        "strip_accents": null,
        "lowercase": true
      }
    ]
  }
}

Note: There is a known issue where the Replace normalizer expects the value of pattern to be a regex and not an object with the regex. It'll be fixed soon.

Pretokenizer

This step splits the input text into tokens. You can use Whitespace for this step since it aligns with your cleaning process(splitting the text based on spaces).

"pre_tokenizer": {
  "type": "Whitespace"
}

Model

Since you used TfidfVectorizer, this handles the tokenization and vectorization, which isn’t directly compatible with tokenizers like WordPiece or Unigram. But TransformersPHP allows for legacy tokenizers with just a vocab so you could try that

"model": {
  "vocab": tfidf_clean.vocabulary_
}

Post Processor and DEcoder

In your own case, ie a text classification task, these are not neccesary. So,

{
...
"post_processor": null,
"decoder": null
}

All these config should happen in your python script after which you save to file. Here's how the final tokenizer looks

import json

# Save the tokenizer configuration
tokenizer_config = {
    "normalizer": {
        "type": "Sequence",
        "normalizers": [
          {
            "type": "Replace",
            "pattern": {
              "Regex": "<.*?>"
            },
            "content": ""
          },
          {
            "type": "Replace",
            "pattern": {
              "Regex": "\\b(" + "|".join(stop_words) + ")\\b"
            },
            "content": ""
          },
          {
            "type": "BertNormalizer",
            "clean_text": true,
            "handle_chinese_chars": true,
            "strip_accents": null,
            "lowercase": true
          }
        ]
      },
      "pre_tokenizer": {
        "type": "Whitespace"
      },
      "post_processor": null,
      "decoder": null,
      "model": {
        "vocab": tfidf_clean.vocabulary_
      }
    }

tokenizer_path = "build/Arvee/tags-classifier/tokenizer.json"
with open(tokenizer_path, "w") as f:
    json.dump(tokenizer_config, f, indent=2)

I hope this helps

[coogle]

This is great info! Thank you because now the tokenizer.json file makes more sense than it did. Unfortunately though, I am still having problems that started requiring me to hack the library specifically in PretrainedTokenizer (see below). I also saw that you don't support Whitespace as a pre_tokenizer so just to move things alone I switched it to WhitespaceSplit, and that things broke because I also didn't have a tokenizer_config.json file, which I had to create as just an empty JSON object.

Here's what happened when I fixed up the .json files and tried to run the model and the code I had to change before I got stuck:

line 382: $tokens2 was null so I had to change it to:

new PostProcessedOutput(tokens: array_merge($tokens, $tokens2 ?? []));

line 306: $maxLength was null ... I was able to get past the error by passing in a max length calculated from the max length of sequences (e.g. max(array_map(fn($x) => count($x['input_ids']), $encodedTokens)), but not sure if that was even the right fix :))

That all ended up with me still getting this exception:

Codewithkyrian\Transformers\Exceptions\MissingModelInputException The following model inputs are missing: float_input.

With all of this in mind is part of the issue we're having here that I'm just doing things in a kind of weird and odd way when I make my model in the first place (that's what this seems like -- e.g. using tfidf which you described as "legacy"). At the end of the day what I am most interested in here is simply training a model to accept some text input from the user and assign it N relevant tags (including zero). If there is a faster approach that is more compatible with the way Transformers wants to deal with things I'd be willing to do that too (as long as I can find a good example to work off from). Would you happen to have good suggestions on an less-legacy way to build my model that would also be a lot easier to work with in your library?

[coogle]

Wanted to give an update here..

I went back to the beginning and rather than building my own model from scratch, I fine tuned a BERT model with my data set using PyTorch ... from there it was pretty easy to convert into an ONNX and when I used my new model things went a LOT easier for me.... based on the code you provided I was able to get the model working with code such as the following:

            $modelObj = AutoModel::fromPretrained($model);
            $tokenizer = AutoTokenizer::fromPretrained($model);

            $id2label = $modelObj->config->config['id2label'];

            $modelInputs = $tokenizer->tokenize($input, padding: true, truncation: true);
            $outputs = $modelObj($modelInputs);

            $retval = [];

            $scores = $outputs['logits']->toArray()[0];

            foreach($scores as $id => $score)
            {
                if($score > 0)
                {
                    $retval[] = $id2label[$id];
                }
            }

            return $retval;

I couldn't figure out how to use the pipeline directly as you suggested -- if there is an easier way to get from scores to the labels baked into the code base already that I can use please let me know.

That said, I got it working!

[CodeWithKyrian]

I was going to suggest fine tuning a smaller Bert model like Albert or Distilbert but seems you've taken care of that. Nice!

In that case, provided you saved the configs et al, the pipeline helper function is super compatible with it as shown on the docs. Did you try that?

[coogle]

Yes, but the problem I ran into is I'm looking for multi-label outputs... so for example when I try:

$classifier = pipeline('text-classification', $model);
$classifier($input, multiLabel: true);

I get one result when I should get two. When I try zero-shot, it throws an exception:

   TypeError  str_replace(): Argument #2 ($replace) must be of type array|string, null given.

--
 () at vendor/codewithkyrian/transformers/src/Pipelines/ZeroShotClassificationPipeline.php:99

Honestly it's not a big deal for me that I had to use a more verbose path vs. just using the pipeline helper -- I just cleaned up my outputs with a sigmoid and applied a threshold of > 0.5 which worked well -- probably shouldn't throw that exception though when I tried to use it as a zero-shot :)

[CodeWithKyrian]

I'm trying to understand what you mean by multi-label outputs. Normally, the pipeline returns only the top label. If you'd like to return the scores for all labels, or a specific number of labels, you can pass a topK argument into the pipeline invocation as indicated in the docs.

$classifier($input, topK: 3);

Setting topK to null returns all labels and their scores.

Regarding multi-label classification in text classification, it works similarly to the multiLabel of zero-shot classification. For the classification pipeline to use multi-label, the code looks for a problem_type key in the model config set to multi_label_classification.

You may wonder why I didn't expose a multiLabel argument similar to the zero-shot classification pipeline to force multi-label classification when invoking the model. The reason is that zero-shot classification models are generally more versatile and handle classifying based on labels they were not explicitly trained to use. Not all models are suitable for zero-shot classification. For example, your model wouldn't work for zero-shot classification since it lacks the necessary configurations.

Text classification models are mostly fine-tuned for a specific set of labels. If they are to be used for multi-label scenarios, they require additional training to function properly in such contexts, which is why this has to be indicated in the model config. While it's possible to expose the multiLabel argument to the text classification pipeline, forcing it on models not fine-tuned for multi-label classification could lead to performance degradation.

Actually, if you look into TextClassificationPipeline, the only difference for multi-label is that I use a sigmoid function to scale the probabilities between 0 and 1, while for single-label, I use softmax. I hope this explanation clarifies things for you!

[coogle]

What I mean by multi-label outputs is I've got a training data set that maps input text to a few hundred different labels/tags, and generally speaking every input text has multiple labels it should be assigned.

That being said, I looked into this again and I think in my juggling around with half-baked code I was wrong to say the pipeline function wasn't doing as I asked if I specify the topK argument:

collect($classifier('Text to extract tags from', topK: 5))->filter(fn($x) => $x['score'] > 0.5)->toArray();

Works as expected for my dataset (I get 5 results back, of descending order of scores).

if I try to pass a topK value of -1 I get an error:

InvalidArgumentException Invalid shape numbers. It gives -1.

The way I worked around this (because I don't know what my topK value should be, I want all the tags that exceed the threshold) is to look at all the logit outputs from the model normalized with sigmoid and return all that are above the desired threshold. This is probably analogous to setting a topK value to -1 (if that was working for me).

            $modelObj = AutoModel::fromPretrained($model);
            $tokenizer = AutoTokenizer::fromPretrained($model);

            $id2label = $modelObj->config->config['id2label'];

            $modelInputs = $tokenizer->tokenize($input, padding: true, truncation: true);
            $outputs = $modelObj($modelInputs);

            $retval = [];

            $scores = Math::sigmoid($outputs['logits']->toArray()[0]);

            foreach ($scores as $id => $score) {
                if ($score >= $threshold) {
                   // Record match in $retval
                }
            }