Open Julia-Penfield opened 3 years ago
Thanks for the report, we'll take a look at that!
To be clear, this is a model that you trained in 3.1.2 initially, and then resumed training on? Or did you initially train this with 3.0.6 and then try to resume in 3.1.2?
Also if you could go ahead and provide the whole stack trace that'd be helpful. You can put it behind a <details>
tag so it's hidden by default.
Hi @polm,
Thanks for the reply.
The model that I am working on and reported the issue about its continued training is only trained in 3.1.2 - both the initial and the continuation attempt. I have not tried resuming training of a model that is initially trained in 3.0.6, but I will likely give it a try soon.
There are two separate cases that I put here.
If the config file is setup as:
[components.ner] source = "./saved_model/model-last" component = "ner"
[components.tok2vec] source = "./saved_model/model-last" component = "tok2vec"
In this case, the stack trace is short:
[2021-09-13 00:42:10,986] [INFO] Set up nlp object from config [2021-09-13 00:42:10,996] [INFO] Pipeline: ['tok2vec', 'ner'] [2021-09-13 00:42:10,996] [INFO] Resuming training for: ['ner', 'tok2vec'] [2021-09-13 00:42:11,000] [INFO] Created vocabulary /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/spacy/training/initialize.py:137: UserWarning: [W113] Sourced component 'tok2vec' may not work as expected: source vectors are not identical to current pipeline vectors. warnings.warn(Warnings.W113.format(name=sourced_component)) /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/spacy/training/initialize.py:137: UserWarning: [W113] Sourced component 'ner' may not work as expected: source vectors are not identical to current pipeline vectors. warnings.warn(Warnings.W113.format(name=sourced_component)) [2021-09-13 00:42:11,001] [INFO] Finished initializing nlp object [2021-09-13 00:42:11,001] [INFO] Initialized pipeline components: []
ℹ Saving to output directory: saved_model ℹ Using CPU
=========================== Initializing pipeline =========================== ✔ Initialized pipeline
============================= Training pipeline ============================= ℹ Pipeline: ['tok2vec', 'ner'] ℹ Initial learn rate: 0.001 E # LOSS TOK2VEC LOSS NER ENTS_F ENTS_P ENTS_R SCORE
If the config file is setup like this:
[components.ner] source = "./saved_model/model-last"
[components.tok2vec] source = "./saved_model/model-last"
Then the stack trace is very long as shown below.
Hi @polm,
Just a mini-update that I down-versioned my spaCy and tried the same thing with 3.0.6 and it worked successfully. So I can confirm that the issue of continuation of training is a problem related to an upgrade after 3.0.6.
Cheers, Julia
Thanks for the extra detail! Sorry it's taking us a while to get to this - we haven't forgotten about it.
I am trying to continue training an NER model. I have done this in spaCy 3.0.6 in the past numerous times, but after coming back to do it again, I realized that this time it's not working. My current version is 3.1.2.
My approach to continued training:
To continue training, I basically modified the config file. The 2 components below are changed to the ones following:
Initial config file setup:
Modified config file:
"./saved model" is the address to which the initial training is saved.
I also tried with the following config file setups:
This approach seem to not work anymore as I get the following error:
Question: Has something changed in the recent versions to disable this approach?
I can send the backtrace if it could be helpful, but beware that it's super long and I could not learn the issue from it. :)
I would appreciate any recommendations. I can try in spaCy 3.0.6 if it helps to verify that it's a version issue.
My Environment