Error with model.predict() when using early stopping

[swtive]

Hi, I trained my model using this and used early stopping criterian in setfit model. After training saved the model as : model.save_pretrained("setfit_early_stopping") and then I am trying to load the model and predict on my text, using below code: model = SetFitModel.from_pretrained("setfit_early_stopping") model.predict([text]) I am getting error of "This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator."

[tomaarsen]

Hello!

Can you check whether the "setfit_early_stopping" directory has a "model_head.pkl" file? Perhaps it wasn't correctly saved, meaning that it just loaded a non-fitted LogisticRegression head instead.

• Tom Aarsen

[swtive]

yes, model_head.pkl file is present in the folder of "setfit_early_stopping". Is this the right way to save the model for its further use : model.save_pretrained("setfit_early_stopping") ?

[tomaarsen]

Apologies for my late response. That is odd behaviour, let me try and reproduce this. model.save_pretrained("setfit_early_stopping") is indeed correct, and so is model = SetFitModel.from_pretrained("setfit_early_stopping"). At a glance it seems like you did everything correctly.

• Tom Aarsen

[tomaarsen]

I'm struggling to reproduce this using the v1.0.0-pre branch. Perhaps I unknowingly resolved the issue in that branch. If you want, you can try to provide a more complete snippet to reproduce the issue.

• Tom Aarsen

[syx1031]

I also found out this problem! And even without early-stop, just model.from_pretrained("Path_to_my_local_checkpoint") and then used model.predict. I'm sure that checkpoint is correct and complete.

[lyons7]

I'm running into the same issue as @syx1031 ! No early stopping, did everything according to the example but get a NotFittedError: This LogisticRegression instance is not fitted yet error with model.predict after loading in the model from the checkpoints/step_28000 folder. (I run into the same issue with model(['example sentence', 'example sentence']))

setfit version is 1.0.3

Reviewing my code, for some reason I didn't pass TrainingArguments to the Trainer -- could this be the culprit? I followed the TrainingArguments in the example, but I think if I didn't actually pass those along, it went with the default which is NOT to save the best model. I'm thinking this could be why? Trying it out now with args=args...

[drbrownman]

Ran into this today. Storing and loading the model head explicitly fixes this.

To store the model head after training pickle.dump(model.model_head, open(modelHeadPath, 'wb'))

To load the saved model head

with open(modelHeadPath, 'rb') as mhf:
    inferenceModel.model_head = pickle.load(mhf)

[xanderex-sid]

# Extract embeddings from the SetFitModel's body
train_embeddings = model.model_body.encode([data['text'] for data in train_dataset])

# Fit the LogisticRegression head
model.model_head.fit(train_embeddings, [data['label'] for data in train_dataset])

This code worked for me.

[drjct]

Ran into this as well on SetFit 1.1.0. Even though model_head.pkl is present after running SetFitModel.from_pretrained()

@drbrownman's fix worked for me. Seems like the incorrect model_head.pkl is being saved out from SetFitModel.from_pretrained()