ConstructorError when try to trainer.model._save_pretrained

[LukasKriesch]

Hi,

I used setfit for training a model. When I want to save it calling trainer.model._save_pretrained(path) I receive the following error:

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar' in "<unicode string>", line 52, column 14: value: !!python/object/apply:numpy.core ... ^ Thanks!

[tomaarsen]

Hello!

Saving a SetFitModel should be done with trainer.model.save_pretrained(path). The _save_pretrained method is just used internally.

However, your error seems a bit unrelated, so it may still occur. It seems like SetFit is trying to write a model card with a numpy scalar when it just expected e.g. a float or a string. Do you perhaps use a custom metric? Or anything else non-standard perhaps?

• Tom Aarsen

[LukasKriesch]

Hi, yes, I use a custom metric for evaluating the model: def compute_metrics(y_pred, y_test): accuracy = accuracy_score(y_test, y_pred) precision = precision_score(y_test, y_pred) recall = recall_score(y_test, y_pred) f1 = f1_score(y_test, y_pred) return { 'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

EDIT: Tried it without a custom metric, than it works. Is there anything I need to adjust with the metric function?

[tomaarsen]

Yes, for the time being, please convert the values in the dictionary to floats, and it should work again! Apologies for the inconvenience. For information, SetFit now creates model cards using the computed metrics, but it seems like it expects evaluate to always return a dictionary with string keys and integer/float/string values, i.e. nothing numpy or torch. I'll have to prepare a patch for this.

• Tom Aarsen