TST Transformers tutorial - error while exporting the model

[ninoslavc]

I went through the tutorial: 07_Time_Series_Classification_with_Transformers.ipynb

Everything worked fine till I tried to export the TST model(s):

PATH = Path('models/Transformers.pkl')
learn.export(PATH)

caused TypeError: cannot pickle 'code' object

My configuration is following:

os              : Windows-10-10.0.19044-SP0
python          : 3.9.13
tsai            : 0.3.4
fastai          : 2.7.10
fastcore        : 1.5.27
sktime          : 0.15.0
sklearn         : 1.1.3
torch           : 1.13.1+cu117
device          : 1 gpu (['NVIDIA GeForce RTX 3060'])
cpu cores       : 8
threads per cpu : 2
RAM             : 31.65 GB
GPU memory      : [12.0] GB

I tried to export the inception model at the beginning of the same tutorial, and there were no problems. I tried my own TST Transformers model and had the same problem. I have no problems exporting other tsai models in other tutorials.

How can this problem be solved? Also, I would suggest adding the part with inference on new data in this tutorial, like in others.

[oguiza]

Hi @ninoslavc, I've checked this code and it works:

PATH = Path('models')
PATH.mkdir(parents=True, exist_ok=True)
learn.export(PATH/'Transformer.pkl')
learn2 = load_learner(PATH/'Transformer.pkl')

Please, try it and let me know if you still have the issue.

[ninoslavc]

Hi @oguiza

Thank you. Unfortunately, I still have the same problem. Here is the full traceback:

TypeError                                 Traceback (most recent call last)
Cell In[56], line 3
      1 PATH = Path('models')
      2 PATH.mkdir(parents=True, exist_ok=True)
----> 3 learn.export(PATH/'Transformer.pkl')
      4 learn2 = load_learner(PATH/'Transformer.pkl')

File ~\AppData\Roaming\Python\Python39\site-packages\fastai\learner.py:428, in export(self, fname, pickle_module, pickle_protocol)
    425 with warnings.catch_warnings():
    426     #To avoid the warning that come from PyTorch about model not being checked
    427     warnings.simplefilter("ignore")
--> 428     torch.save(self, self.path/fname, pickle_module=pickle_module, pickle_protocol=pickle_protocol)
    429 self.create_opt()
    430 if state is not None: self.opt.load_state_dict(state)

File ~\AppData\Roaming\Python\Python39\site-packages\torch\serialization.py:423, in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
    421 if _use_new_zipfile_serialization:
    422     with _open_zipfile_writer(f) as opened_zipfile:
--> 423         _save(obj, opened_zipfile, pickle_module, pickle_protocol)
    424         return
    425 else:

File ~\AppData\Roaming\Python\Python39\site-packages\torch\serialization.py:635, in _save(obj, zip_file, pickle_module, pickle_protocol)
    633 pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)
    634 pickler.persistent_id = persistent_id
--> 635 pickler.dump(obj)
    636 data_value = data_buf.getvalue()
    637 zip_file.write_record('data.pkl', data_value, len(data_value))

TypeError: cannot pickle 'code' object

Unusually, I have that problem only with TST models.

[oguiza]

Hi @ninoslavc, There's something else you can try: remove the callbacks before exporting the model. I've seen that callbacks can sometimes create issues.

PATH = Path('models')
PATH.mkdir(parents=True, exist_ok=True)
learn.remove_cbs(learn.cbs) # <--- INSERT THIS
learn.export(PATH/'Transformer.pkl')
learn2

[ninoslavc]

Hi @oguiza,

Unfortunately, that did not help:

TypeError                                 Traceback (most recent call last)
Cell In[30], line 4
      2 PATH.mkdir(parents=True, exist_ok=True)
      3 learn.remove_cbs(learn.cbs) # <--- INSERT THIS
----> 4 learn.export(PATH/'Transformer.pkl')
      5 learn2

File ~\AppData\Roaming\Python\Python39\site-packages\fastai\learner.py:428, in export(self, fname, pickle_module, pickle_protocol)
    425 with warnings.catch_warnings():
    426     #To avoid the warning that come from PyTorch about model not being checked
    427     warnings.simplefilter("ignore")
--> 428     torch.save(self, self.path/fname, pickle_module=pickle_module, pickle_protocol=pickle_protocol)
    429 self.create_opt()
    430 if state is not None: self.opt.load_state_dict(state)

File ~\AppData\Roaming\Python\Python39\site-packages\torch\serialization.py:423, in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
    421 if _use_new_zipfile_serialization:
    422     with _open_zipfile_writer(f) as opened_zipfile:
--> 423         _save(obj, opened_zipfile, pickle_module, pickle_protocol)
    424         return
    425 else:

File ~\AppData\Roaming\Python\Python39\site-packages\torch\serialization.py:635, in _save(obj, zip_file, pickle_module, pickle_protocol)
    633 pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)
    634 pickler.persistent_id = persistent_id
--> 635 pickler.dump(obj)
    636 data_value = data_buf.getvalue()
    637 zip_file.write_record('data.pkl', data_value, len(data_value))

TypeError: cannot pickle 'code' object

[oguiza]

Hi again @ninoslavc, I'm running out of options. The issue is that I cannot reproduce the issue. See the code working in Colab: gist I'm not sure if this will make a difference, but you could pass the path when you instantiate the learner. Pass it as path:

from tsai.models.TST import TST
X, y, splits = get_UCR_data('LSST', split_data=False)
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
PATH = Path('models')
PATH.mkdir(parents=True, exist_ok=True)
learn = ts_learner(dls, TST, metrics=accuracy, path=PATH, cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)
learn.export('Transformer.pkl')

Let me know if this doesn't work.

BTW, do you create any function/class that ends up in the learner in your script? Or are all functions/ classes in modules?

[ninoslavc]

Hi @oguiza, Thank you for your suggestions. I did not change anything in the tutorial file - I used it as is. Adding a path did not help. Also, I tried to change cbs=ShowGraphCallback2() with cbs=[ShowGraph()], but still got the error TypeError: cannot pickle 'code' object. I don't get how come that only TST models have that problem. All others work fine.

[oguiza]

ok. Are you using loss_func=LabelSmoothingCrossEntropyFlat() in your learner. If so, please, remove it, and try it again. I had a similar pickle issue last year. I'm just trying to isolate the issue. You may try removing certain components(like ShowGraph()) that are not necessary to run the code, and see if you can find what's causing the issue.

[ninoslavc]

Hi @oguiza, you are right, loss_func=LabelSmoothingCrossEntropyFlat() caused the problem. After removing it, saving it to .pkl works fine.

Thank you