self supervised BERT

[geoHeil]

I try to follow along with:

However, it fails with: ModuleAttributeError: 'InceptionTimePlus' object has no attribute 'TSBERT' :

---------------------------------------------------------------------------
ModuleAttributeError                      Traceback (most recent call last)
<ipython-input-23-1b09c764264b> in <module>
----> 1 learn.TSBERT.show_preds(sharey=True)

/path/to/myenv/lib/python3.8/site-packages/fastcore/basics.py in __getattr__(self, k)
    386         if self._component_attr_filter(k):
    387             attr = getattr(self,self._default,None)
--> 388             if attr is not None: return getattr(attr,k)
    389         raise AttributeError(k)
    390     def __dir__(self): return custom_dir(self,self._dir())

/path/to/myenv/lib/python3.8/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    776             if name in modules:
    777                 return modules[name]
--> 778         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
    779             type(self).__name__, name))
    780 

ModuleAttributeError: 'InceptionTimePlus' object has no attribute 'TSBERT'

Even though https://github.com/timeseriesAI/tsai/blob/main/tutorial_nbs/08_Self_Supervised_TSBERT.ipynb

learn = construct_learner(udls100, InceptionTimePlus, cbs=[ShowGraph(), TSBERT(target_dir='./data/TSBERT', fname=f'{dsid}_200')])

learn.fit_one_cycle(5, 3e-3)
learn.TSBERT.show_preds(sharey=True) # this fails!

somehow apparently must work in the example notebook.

I am really confused as I basically follow the same steps 1:1 but get this error.

The versions used on my machine:

tsai       : 0.2.18
fastai     : 2.3.1
fastcore   : 1.3.19
torch      : 1.7.1

[oguiza]

Hi, The issue is that TSBERT to renamed to MVP (for Masked-value predictor) as it can be used by any type of model, not just transformers as the BERT indicates. If you change the line to:

learn.MVP.show_preds(sharey=True)

it should work.

Edit: For compatibility, I've also added the option to use

learn.TSBERT.show_preds(sharey=True)

so this line should also work now

[geoHeil]

great / thanks.

[geoHeil]

too early. for MWDNPlus I get Exception: No shared weight names were found between the models after training it for quite some while.

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-27-dd1ccb302bfe> in <module>
----> 1 learn = ts_learner(dls010, current_learner_model, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=current_metrics)
      2 for p in learn.model.parameters():
      3     p.requires_grad=False
      4 print(f'{"trainable params once manually frozen":40}: {count_parameters(learn.model):8}')
      5 learn.freeze()

path/to/conda/env_foo/lib/python3.8/site-packages/tsai/learner.py in ts_learner(dls, arch, c_in, c_out, seq_len, d, splitter, loss_func, opt_func, lr, cbs, metrics, path, model_dir, wd, wd_bn_bias, train_bn, moms, **kwargs)
    222 
    223     if arch is None: arch = InceptionTimePlus
--> 224     model = build_ts_model(arch, dls=dls, c_in=c_in, c_out=c_out, seq_len=seq_len, d=d, **kwargs)
    225     try:
    226         model[0], model[1]

path/to/conda/env_foo/lib/python3.8/site-packages/tsai/models/utils.py in build_ts_model(arch, c_in, c_out, seq_len, d, dls, device, verbose, pretrained, weights_path, exclude_head, cut, init, **kwargs)
    160     if pretrained and not ('xresnet' in arch.__name__ and not '1d' in arch.__name__):
    161         assert weights_path is not None, "you need to pass a valid weights_path to use a pre-trained model"
--> 162         transfer_weights(model, weights_path, exclude_head=exclude_head, device=device)
    163 
    164     if init is not None:

path/to/conda/env_foo/lib/python3.8/site-packages/tsai/models/utils.py in transfer_weights(model, weights_path, device, exclude_head)
    103             unmatched_layers.append(name)
    104             pass # these are weights that weren't in the original model, such as a new head
--> 105     if matched_layers == 0: raise Exception("No shared weight names were found between the models")
    106     else:
    107         if len(unmatched_layers) > 0:

Exception: No shared weight names were found between the models

It turns out this does not happen in the case of inception time plus

[oguiza]

Hi @geoHeil, I'm not sure what the issue is. I've tested the code in a gist and it works well. Are you sure the weights_path is the one printed out by MVP when you pretrained the model?

[geoHeil]

indeed - you are correct. Sorry for the mixup!