search

facebookresearch / SpanBERT

Code for using and evaluating SpanBERT.
Other
884 stars 174 forks source link

How should i use train.py to load a SpanBERT model and continue pretraining it? #72

Open WorldWarII opened 3 years ago

WorldWarII commented 3 years ago

I tried to use train.py load the model you gave(SpanBERT base & cased) and continue pretraining it on my own medicial corpus.I added the argument '--restore-file pytorch_model.bin', and changed the argument -- arch from 'cased_bert_pair_large' to 'cased_bert_pair', but i find it can't be loaded this way. the KeyError shows that 'state' don't have a key names 'best_loss'. I looked into 'save_state' method in pretraining/utils.py, and found these codes: state_dict = { 'args': args, 'model': model.state_dict() if model else {}, 'optimizer_history': optim_history + [ { 'criterion_name': criterion.__class__.__name__, 'optimizer_name': optimizer.__class__.__name__, 'lr_scheduler_state': lr_scheduler.state_dict(), 'num_updates': num_updates, } ], 'last_optimizer_state': convert_state_dict_type(optimizer.state_dict()), 'extra_state': extra_state, } torch_persistent_save(state_dict, filename) It seems that you saved more than just the state_dict of the model during pretraining.And I still cant see where 'best_loss' is. Can I get a checkpoint fire which can be load by train.py？Or how to load the fire you released by train.py?

houliangxue commented 2 years ago

hi，I am facing the same problem. Is there anything solution, please

WorldWarII commented 2 years ago

hi，I am facing the same problem. Is there anything solution, please

It has been too long and I am not sure now. But it seems that the checkpoint file to be loaded by the training code is different from the file author provided, you may have to build a checkpoint file by yourself.