tomaarsen
commented
1 year ago Hello @fpservant,
I believe this is because accuracy from the Hugging Face's evaluate package expects the ground truth to be integer labels, rather than strings:
>>> import evaluate
>>> accuracy_metric = evaluate.load("accuracy")
>>> accuracy_metric.compute(references=[0, 1], predictions=[1, 1])
{'accuracy': 0.5}
>>> accuracy_metric.compute(references=["a", "b"], predictions=["b", "b"])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "[sic]\evaluate\module.py", line 432, in compute
self.add_batch(**inputs)
File "[sic]\evaluate\module.py", line 486, in add_batch
batch = self.selected_feature_format.encode_batch(batch)
File "[sic]\datasets\features\features.py", line 1596, in encode_batch
encoded_batch[key] = [encode_nested_example(self[key], obj) for obj in column]
File "[sic]\datasets\features\features.py", line 1596, in <listcomp>
encoded_batch[key] = [encode_nested_example(self[key], obj) for obj in column]
File "[sic]\datasets\features\features.py", line 1203, in encode_nested_example
return schema.encode_example(obj) if obj is not None else None
File "[sic]\datasets\features\features.py", line 465, in encode_example
return int(value)
ValueError: invalid literal for int() with base 10: 'b'This can be resolved by encoding your labels, e.g.:
# Apply encoding
label_to_int = {label: int for int, label in enumerate(df["label"].unique())}
# {'LABEL1': 0, 'LABEL2': 1, 'LABEL3': 2}
df["label"] = df["label"].map(label_to_int)(See full working script ready to copy-paste)
```python from sentence_transformers.losses import CosineSimilarityLoss from setfit import SetFitModel, SetFitTrainer from datasets import Dataset import pandas as pd df = pd.DataFrame([ ['text 1','LABEL1'], ['text 10','LABEL1'], ['text 11','LABEL1'], ['text 12','LABEL1'], ['foo 2','LABEL2'], ['foo 20','LABEL2'], ['foo 21','LABEL2'], ['foo 22','LABEL2'], ['bar 3','LABEL3'], ['bar 30','LABEL3'], ['bar 31','LABEL3'], ['bar 32','LABEL3'], ]) df.columns = ['text','label'] # Apply encoding label_to_int = {label: int for int, label in enumerate(df["label"].unique())} # {'LABEL1': 0, 'LABEL2': 1, 'LABEL3': 2} df["label"] = df["label"].map(label_to_int) train_ds = Dataset.from_pandas(df) eval_ds = train_ds # yes, OK, normally not what we would do model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2") trainer = SetFitTrainer( model=model, train_dataset=train_ds, eval_dataset=eval_ds, loss_class=CosineSimilarityLoss, metric="accuracy", batch_size=4, num_iterations=3, # The number of text pairs to generate for contrastive learning num_epochs=1, # The number of epochs to use for constrastive learning ) trainer.train() print(trainer.evaluate()) ```This now outputs:
model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.
***** Running training *****
Num examples = 72
Num epochs = 1
Total optimization steps = 18
Total train batch size = 4
Iteration: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:06<00:00, 2.99it/s]
Epoch: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00, 6.02s/it]
***** Running evaluation *****
{'accuracy': 1.0}I don't believe that this is mentioned in the README, and we don't currently have any docs set up. Apologies for this. When docs are being set up, this certainly should be included. Alternatively, we could implement an encoding behind the scenes as a pre-processing step before training when we encounter string labels.
Hope this helps somewhat.
- Tom Aarsen
fpservant
AppleMax1992
anth0nyhak1m
vbabashov
ValueError in trainer.evaluate in following conditions : Dataset created from a pandas DataFrame, label column containing strings.
Here is what I do:
Here's the output of last line:
Best Regards, fps