search

yzhangcs / parser

:rocket: State-of-the-art parsers for natural language.
https://parser.yzhang.site/
MIT License
837 stars 143 forks source link

How to retrain a pretrained model using checkpoint function #102

Closed lmompela closed 2 years ago

lmompela commented 2 years ago

I have trained a first model using camemBERT and with train, test, and dev files on French. Once the task is completed, I would like to use the resulting model and retrain it on different sets of data, specifically using data from another language, to finetune the model.

I have tried using the checkpoint function, but as soon as biaffine finds the best model, it doesn't allow another round of training using the checkpoint function. Instead, it prints this message : """ Traceback (most recent call last): File "/N/soft/rhel7/python/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code(code, main_globals, None, File "/N/soft/rhel7/python/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code exec(code, run_globals) File "/N/u/lmompela/Carbonate/.local/lib/python3.8/site-packages/supar/cmds/biaffine_dep.py", line 47, in main() File "/N/u/lmompela/Carbonate/.local/lib/python3.8/site-packages/supar/cmds/biaffine_dep.py", line 43, in main parse(parser) File "/N/u/lmompela/Carbonate/.local/lib/python3.8/site-packages/supar/cmds/cmd.py", line 28, in parse parser = Parser.load(args) if args.checkpoint else Parser.build(args) File "/N/u/lmompela/Carbonate/.local/lib/python3.8/site-packages/supar/parsers/dep.py", line 152, in load return super().load(path, reload, src, **kwargs) File "/N/u/lmompela/Carbonate/.local/lib/python3.8/site-packages/supar/parsers/parser.py", line 203, in load parser.checkpoint_state_dict = state['checkpoint_state_dict'] if args.checkpoint else None KeyError: 'checkpoint_state_dict'

""" I know it's not due to the bash file I used since it's the same I used to start training from a previous checkpoint (which worked then since biaffine hadn't found the best model yet before the training was halted)

Do you know whether this is possible?

yzhangcs commented 2 years ago

@lmompela Hi, to retrain the model you might need to revise some code to support loading state dict from the previous trained one. However, it's not possible to reload the states of previous scheduler/optimizer as they will be discarded once the training process is completed.

lmompela commented 2 years ago

Hi, thank you for the quick reply! Ok I see, so just to make sure I understand, there is for now no way to train a model on one language and retrain it on another language without having to touch the code at all?

If not, then I'll try and see if I can tweak some code, and this issue can be closed. Thanks!

yzhangcs commented 2 years ago

@lmompela Yes, and I don't recommend this way as it may cause the model to forget what it has previously learned.