MAIF / melusine

📧 Melusine: Use python to automatize your email processing workflow
https://maif.github.io/melusine
Other
352 stars 58 forks source link

fix issue when saving Bert model #118

Closed CRGONCALVES closed 2 years ago

CRGONCALVES commented 2 years ago

I have an issue when saving the Bert Model with the code below :

import joblib
_ = joblib.dump(nn_model,"./data/nn_model.pickle",compress=True)

I fix this issue by adding the code : if "embedding_matrix"] in dict_attr: in the file train.py (see below) :

def __getstate__(self):
        """Method called before serialization for a specific treatment to save
        model weight and structure instead of standard serialization."""
        dict_attr = dict(self.__dict__)
        if "model" in dict_attr:
            del dict_attr["model"]
            if "embedding_matrix"] in dict_attr:
                del dict_attr["embedding_matrix"]
            del dict_attr["pretrained_embedding"]
        return dict_attr

Test Configuration:

CRGONCALVES commented 2 years ago

import joblib _ = joblib.dump(CamemBert_model,"./data/CamemBert_model.pickle",compress=True)

hugo-quantmetry commented 2 years ago

Hello Carlos, Thank you for your contribution! I added some comments:

TFA-MAIF commented 2 years ago

Fixed PR has been merge.