Evaluation not working with `microsoft/deberta-v3-large`

[msinha251]

Hi,

I was trying to evaluate the model trained on microsoft/deberta-v3-large but looks like it's having some issues on indexing. Any ideas or support for this one ?

Error: IndexError: list index out of range

Code to Reproduce:

import pandas as pd
from datasets import load_dataset
from evaluate import evaluator
from transformers import pipeline

models = [
    "microsoft/deberta-v3-large"
]

data = load_dataset("conll2003", split="validation").shuffle().select(range(10))
task_evaluator = evaluator("token-classification")

results = []
for model in models:
    results.append(
        task_evaluator.compute(
            model_or_pipeline=model, data=data, metric="seqeval"
            )
        )

df = pd.DataFrame(results, index=models)
df[["overall_f1", "overall_accuracy", "total_time_in_seconds", "samples_per_second", "latency_in_seconds"]]

[msinha251]

Can someone please advice, it's still the same issue ?

[maayanorner]

Same issue, another model.