saved models

[attardi]

It seems that the models saved with torch.save() include external objects, like BertTokenizer. If you try to run the model on a machine where a new version of transformers (e.g. 3.1.0) becomes available, the program will crash. This is a pity, since it makes all trained model no more usable. It should be better to avoid saving the whole tokenizer object and only save its class name in order to recreate a new instance when loading the model.

python -m supar.cmds.biaffine_dependency predict -p=exp/fr-bert/model --tree \ --data=fr.conllu --pred=/dev/null

2020-09-18 17:00:10 INFO Loading the data Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/homenfs/tempGPU/iwpt2020/supar/supar/cmds/biaffine_dependency.py", line 43, in main() File "/homenfs/tempGPU/iwpt2020/supar/supar/cmds/biaffine_dependency.py", line 39, in main parse(parser) File "/homenfs/tempGPU/iwpt2020/supar/supar/cmds/cmd.py", line 35, in parse parser.predict(args) File "/homenfs/tempGPU/iwpt2020/supar/supar/parsers/biaffine_dependency.py", line 125, in predict return super().predict(Config().update(locals())) File "/homenfs/tempGPU/iwpt2020/supar/supar/parsers/parser.py", line 137, in predict dataset.build(args.batch_size, args.buckets) File "/homenfs/tempGPU/iwpt2020/supar/supar/utils/data.py", line 88, in build self.fields = self.transform(self.sentences) File "/homenfs/tempGPU/iwpt2020/supar/supar/utils/transform.py", line 39, in call pairs[f] = f.transform([getattr(i, f.name) for i in sentences]) File "/homenfs/tempGPU/iwpt2020/supar/supar/utils/field.py", line 302, in transform for seq in sequences] File "/homenfs/tempGPU/iwpt2020/supar/supar/utils/field.py", line 302, in for seq in sequences] File "/homenfs/tempGPU/iwpt2020/supar/supar/utils/field.py", line 301, in sequences = [[self.preprocess(token) for token in seq] File "/homenfs/tempGPU/iwpt2020/supar/supar/utils/field.py", line 157, in preprocess sequence = self.tokenize(sequence) File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/transformers/tokenization_utils.py", line 349, in tokenize no_split_token = self.unique_no_split_tokens AttributeError: 'BertTokenizer' object has no attribute 'unique_no_split_tokens'

[yzhangcs]

Hi, thank you for reporting this potential issue. I will fix it soon later, or feel free to create a PR.

[attardi]

I have a patch. In method Parser.load(), it recreates the tokenizer:

    transform = state['transform']
    if args.feat == 'bert':
        tokenizer = SubwordField.tokenizer(args.bert)
        transform.FORM[1].tokenize = tokenizer.tokenize
    return cls(args, model, transform)

invoking this method in class SubwordField:

@classmethod
def tokenizer(cls, name):
    """                                                                                
    Create an instance of tokenizer from either path or name.                          
    :param name: path or name of tokenizer.                                            
    """

    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(name)
    tokenizer.bos_token = tokenizer.bos_token or tokenizer.cls_token
tokenizer.eos_token = tokenizer.eos_token or tokenizer.sep_token
    return tokenizer

and biaffine-parser also uses it to create it, in class method build():

    elif args.feat == 'bert':
        tokenizer = SubwordField.tokenizer(args.bert)

You may also want too avoid to save the tokenize function, by doing this in Parser.save():

    if args.feat == 'bert':
        tokenize = self.transform.FORM[1].tokenize # save it                           
        self.transform.FORM[1].tokenize = None
    state = {'name': self.NAME,
             'args': args,
             'state_dict': state_dict,
             'pretrained': pretrained,
             'transform': self.transform}
    torch.save(state, path)
    if args.feat == 'bert':
        self.transform.FORM[1].tokenize = tokenize # restore                           

What do you think?

I cannot make a PR, since I have other changes in my code.

[yzhangcs]

@attardi That's might not be very elegant, I suppose tokenize to be a more general method, and directly moving the initialization of BertTokenizer inside SubwordField might not be a good choice. I haven't figured out any solution to the above problem though. BTW, with more parsers to be added, I think I'm increasingly losing the control of the field APIs. While torchtext is on the way to redesign their APIs to be more compatible with PyTorch and transformers and the new release can be available in October, I'm considering whether it is worth to replacing some of my impls with torchtext. Do you have any suggestions?

[yzhangcs]

@attardi As a temporary solution, I have re-uploaded the models trained with transformers 3.2.0.

[yzhangcs]

@attardi Done. The newly uploaded models are trained with transformers==3.2.0, and can be loaded with transformers==4.0.0.