Classify endpoint not available for a finetuned DebertaV2ForSequenceClassification model

[dblakely]

System Info

Running infinity via docker (michaelf34/infinity:latest) + using the REST API to call the model

Information

• [X] Docker
• [ ] The CLI directly via pip

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

import requests

url = "http://0.0.0.0:2230/classify"
model = "/path/to/model"
payload = {"model": model, "input": ["Munich is in Germany", "The sky is blue"]}

output = requests.post(url, json=payload)

print(output.json())

Which gives me:

{'error': {'message': "ModelNotDeployedError: model=`/path/to/model` does not support `classify`. Reason: the loaded moded cannot fullyfill `classify`.options are {'rerank'}.", 'type': None, 'param': None, 'code': 400}}

The model's config looks like:

{
  "_name_or_path": "microsoft/deberta-v2-xlarge",
  "architectures": [
    "DebertaV2ForSequenceClassification"
  ],
  "attention_head_size": 64,
  "attention_probs_dropout_prob": 0.1,
  "conv_act": "gelu",
  "conv_kernel_size": 3,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1536,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "intermediate_size": 6144,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_eps": 1e-07,
  "max_position_embeddings": 512,
  "max_relative_positions": -1,
  "model_type": "deberta-v2",
  "norm_rel_ebd": "layer_norm",
  "num_attention_heads": 24,
  "num_hidden_layers": 24,
  "pad_token_id": 0,
  "pooler_dropout": 0,
  "pooler_hidden_act": "gelu",
  "pooler_hidden_size": 1536,
  "pos_att_type": [
    "p2c",
    "c2p"
  ],
  "position_biased_input": false,
  "position_buckets": 256,
  "problem_type": "multi_label_classification",
  "relative_attention": true,
  "share_att_key": true,
  "torch_dtype": "float32",
  "transformers_version": "4.34.0",
  "type_vocab_size": 0,
  "vocab_size": 128100
}

So it's specified that the model is for classification in the config.

Expected behavior

While the base deberta-v2 doesn't have a classification head, this checkpoint does and I finetuned it for classification. So I was expecting the /classify endpoint to work in this case. Is there anything wrong with the config or is there a way to have Infinity run a deberta-v2 finetuned for sequence classification?

[michaelfeil]

"id2label": {
    "0": "LABEL_0"
  },

means you only have 1 class -> the above config.json is identical to rerankers.

Loads reranker. Needs at least two trained classes to work with infinity.

[dblakely]

Thanks for the quick reply. So the classification layer is Linear(in_features=1536, out_features=1, bias=True). Is it fine to run this model as a reranker then to get scores? If so, what do you set the query to?

[michaelfeil]

Well, your model is a two class classifier, it bases multi_label_classification but has once classification head dim = 1. My suggestion is is to not create single classification head dim, but use dim=2 for your problem.