search

explosion / spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python
https://spacy.io
MIT License
29.82k stars 4.37k forks source link

Error when training on nightly #6226

Closed fcggamou closed 4 years ago

fcggamou commented 4 years ago

I'm getting this error when trying to train on the nightly version of spacy:

... File "/usr/local/lib/python3.6/dist-packages/thinc/model.py", line 288, in __call__ return self._func(self, X, is_train=is_train) File "/usr/local/lib/python3.6/dist-packages/thinc/layers/softmax.py", line 32, in forward W = cast(Floats2d, model.get_param("W")) File "/usr/local/lib/python3.6/dist-packages/thinc/model.py", line 213, in get_param f"Parameter '{name}' for model '{self.name}' has not been allocated yet." KeyError: "Parameter 'W' for model 'softmax' has not been allocated yet."

How to reproduce the behaviour

!python -m spacy train 'config.cfg' --output='model' --gpu-id=0 --verbose --paths.train train.spacy --paths.dev test.spacy

config.cfg:

`[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "es"
pipeline = ["transformer","tagger","parser","ner"]
tokenizer = {"@<!-- -->tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@<!-- -->architectures = "spacy.TransitionBasedParser.v1"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@<!-- -->layers":"reduce_mean.v1"}

[components.parser]
factory = "parser"
learn_tokens = false
min_action_freq = 30
moves = null
update_with_oracle_cut_size = 100

[components.parser.model]
@<!-- -->architectures = "spacy.TransitionBasedParser.v1"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = false
nO = null

[components.parser.model.tok2vec]
@<!-- -->architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@<!-- -->layers":"reduce_mean.v1"}

[components.tagger]
factory = "tagger"

[components.tagger.model]
@<!-- -->architectures = "spacy.Tagger.v1"
nO = null

[components.tagger.model.tok2vec]
@<!-- -->architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@<!-- -->layers":"reduce_mean.v1"}

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@<!-- -->annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@<!-- -->architectures = "spacy-transformers.TransformerModel.v1"
name = "mrm8488/RuPERTa-base"

[components.transformer.model.get_spans]
@<!-- -->span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@<!-- -->readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@<!-- -->readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 500
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = ["tagger", "parser"]
before_to_disk = null

[training.batcher]
@<!-- -->batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@<!-- -->loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@<!-- -->optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@<!-- -->schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
dep_las_per_type = null
sents_p = null
sents_r = null
ents_per_type = null
tag_acc = 0.33
dep_uas = 0.17
dep_las = 0.17
sents_f = 0.0
ents_f = 0.33
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null

[initialize.components]

[initialize.tokenizer]`

Any clues on what may be happening are appreciated.

Your Environment

svlandeg commented 4 years ago

I'm afraid the nightly is kind of unstable right now, though that will hopefully soon change ;-)

The error looks like the model has not been initialized properly. We recently renamed begin_training to initialize, which should be reflected in the code you have for spacy-transformers, specifically here: https://github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py#L298

Can you check whether your version of spacy-transformers has that fix? Or just pull the latest? There was a bug with is_trainable as well (that needs to return True for the Transformer component, or it won't be updated)

fcggamou commented 4 years ago

Thank you Sofie for your answer.

I indeed have that line in my version: image

Also, I should have mentioned that I'm getting the same error message when trying to train without the spacy-transformers library installed (and with the corresponding generated config file).

svlandeg commented 4 years ago

Could you paste the actual stack trace with the error? Were there any warnings thrown?

svlandeg commented 4 years ago

The fact that it's the softmax layer that 's not properly initialized, makes me think that the output dimension is not properly defined. Are there tagger & NER & parser labels in your data? Are all the components initialized with a proper set of labels?

fcggamou commented 4 years ago

Could you paste the actual stack trace with the error? Were there any warnings thrown?

`2020-10-08 20:02:11.962216: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 ✔ Created output directory: model Config overrides from CLI: ['paths.train', 'paths.dev'] ℹ Using GPU: 0

=========================== Initializing pipeline =========================== Set up nlp object from config Loading corpus from path: test.spacy Loading corpus from path: train.spacy Pipeline: ['transformer', 'tagger', 'parser', 'ner'] Created vocabulary Finished initializing nlp object Downloading: 100% 524/524 [00:00<00:00, 440kB/s] Downloading: 100% 981k/981k [00:00<00:00, 1.89MB/s] Downloading: 100% 523k/523k [00:00<00:00, 1.51MB/s] Downloading: 100% 150/150 [00:00<00:00, 109kB/s] Downloading: 100% 2.00/2.00 [00:00<00:00, 1.56kB/s] Downloading: 100% 507M/507M [00:17<00:00, 28.5MB/s] Initialized pipeline components: ['transformer', 'ner'] ✔ Initialized pipeline

============================= Training pipeline ============================= Loading corpus from path: test.spacy Loading corpus from path: train.spacy ℹ Pipeline: ['transformer', 'tagger', 'parser', 'ner'] ℹ Frozen components: ['tagger', 'parser'] ℹ Initial learn rate: 0.0 E # LOSS TRANSFORMER LOSS NER TAG_ACC DEP_UAS DEP_LAS SENTS_F ENTS_F ENTS_P ENTS_R SCORE

⚠ Aborting and saving the final best model. Encountered exception: "Parameter 'W' for model 'softmax' has not been allocated yet." Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.6/dist-packages/spacy/main.py", line 4, in setup_cli() File "/usr/local/lib/python3.6/dist-packages/spacy/cli/_util.py", line 65, in setup_cli command(prog_name=COMMAND) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.6/dist-packages/typer/main.py", line 497, in wrapper return callback(use_params) # type: ignore File "/usr/local/lib/python3.6/dist-packages/spacy/cli/train.py", line 59, in train_cli train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr) File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 105, in train raise e File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 85, in train for batch, info, is_best_checkpoint in training_step_iterator: File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 201, in train_while_improving score, other_scores = evaluate() File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 252, in evaluate scores = nlp.evaluate(dev_examples) File "/usr/local/lib/python3.6/dist-packages/spacy/language.py", line 1309, in evaluate docs = list(docs) File "spacy/pipeline/transition_parser.pyx", line 168, in pipe File "/usr/local/lib/python3.6/dist-packages/spacy/util.py", line 1322, in minibatch batch = list(itertools.islice(items, int(batch_size))) File "spacy/pipeline/transition_parser.pyx", line 168, in pipe File "/usr/local/lib/python3.6/dist-packages/spacy/util.py", line 1322, in minibatch batch = list(itertools.islice(items, int(batch_size))) File "spacy/pipeline/tagger.pyx", line 122, in pipe File "spacy/pipeline/tagger.pyx", line 140, in spacy.pipeline.tagger.Tagger.predict File "/usr/local/lib/python3.6/dist-packages/thinc/model.py", line 312, in predict return self._func(self, X, is_train=False)[0] File "/usr/local/lib/python3.6/dist-packages/thinc/layers/chain.py", line 54, in forward Y, inc_layer_grad = layer(X, is_train=is_train) File "/usr/local/lib/python3.6/dist-packages/thinc/model.py", line 288, in call return self._func(self, X, is_train=is_train) File "/usr/local/lib/python3.6/dist-packages/thinc/layers/with_array.py", line 41, in forward return _list_forward(cast(Model[List2d, List2d], model), Xseq, is_train) File "/usr/local/lib/python3.6/dist-packages/thinc/layers/with_array.py", line 79, in _list_forward Yf, get_dXf = layer(Xf, is_train) File "/usr/local/lib/python3.6/dist-packages/thinc/model.py", line 288, in call return self._func(self, X, is_train=is_train) File "/usr/local/lib/python3.6/dist-packages/thinc/layers/softmax.py", line 32, in forward W = cast(Floats2d, model.get_param("W")) File "/usr/local/lib/python3.6/dist-packages/thinc/model.py", line 213, in get_param f"Parameter '{name}' for model '{self.name}' has not been allocated yet." KeyError: "Parameter 'W' for model 'softmax' has not been allocated yet."`

fcggamou commented 4 years ago

Are there tagger & NER & parser labels in your data?

In my data there are tagger, NER and parser labels, here is an example:

`[
  {
    "id":0,
    "paragraphs":[
      {
        "raw":"50 a\u00f1os\ndispepsia\ninfecciones urinarias\nprotesis mamaria bilateral\ndejo de tomar psiconor hace 1 mes en forma progresiva\nmotivo de consulta\nrinofaringitis aguda viral\nno fiebre",
        "sentences":[
          {
            "tokens":[
              {
                "id":0,
                "orth":"50",
                "space":" ",
                "tag":"NUM",
                "pos":"NUM",
                "morph":"NumForm=Digit|NumType=Card",
                "lemma":"50",
                "head":1,
                "dep":"nummod",
                "ner":"O"
              },
              {
                "id":1,
                "orth":"a\u00f1os",
                "space":"",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Masc|Number=Plur",
                "lemma":"a\u00f1o",
                "head":11,
                "dep":"nsubj",
                "ner":"O"
              },
              {
                "id":2,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":3,
                "orth":"dispepsia",
                "space":"",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Fem|Number=Sing",
                "lemma":"dispepsia",
                "head":-2,
                "dep":"appos",
                "ner":"U-Term"
              },
              {
                "id":4,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":5,
                "orth":"infecciones",
                "space":" ",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Fem|Number=Plur",
                "lemma":"infecci\u00f3n",
                "head":-4,
                "dep":"appos",
                "ner":"B-Term"
              },
              {
                "id":6,
                "orth":"urinarias",
                "space":"",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Gender=Fem|Number=Plur",
                "lemma":"urinario",
                "head":-1,
                "dep":"amod",
                "ner":"L-Term"
              },
              {
                "id":7,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"Gender=Masc|Number=Sing",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":8,
                "orth":"protesis",
                "space":" ",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Fem",
                "lemma":"protesis",
                "head":-3,
                "dep":"nmod",
                "ner":"B-Term"
              },
              {
                "id":9,
                "orth":"mamaria",
                "space":" ",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Gender=Fem|Number=Sing",
                "lemma":"mamario",
                "head":-1,
                "dep":"amod",
                "ner":"L-Term"
              },
              {
                "id":10,
                "orth":"bilateral",
                "space":"",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Number=Sing",
                "lemma":"bilateral",
                "head":-2,
                "dep":"amod",
                "ner":"O"
              },
              {
                "id":11,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"Gender=Masc|Number=Sing",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":12,
                "orth":"dejo",
                "space":" ",
                "tag":"VERB",
                "pos":"VERB",
                "morph":"Mood=Ind|Number=Sing|Person=3|Tense=Fut|VerbForm=Fin",
                "lemma":"dejar",
                "head":0,
                "dep":"ROOT",
                "ner":"O"
              },
              {
                "id":13,
                "orth":"de",
                "space":" ",
                "tag":"ADP",
                "pos":"ADP",
                "morph":"AdpType=Prep",
                "lemma":"de",
                "head":1,
                "dep":"mark",
                "ner":"O"
              },
              {
                "id":14,
                "orth":"tomar",
                "space":" ",
                "tag":"VERB",
                "pos":"VERB",
                "morph":"VerbForm=Inf",
                "lemma":"tomar",
                "head":-2,
                "dep":"xcomp",
                "ner":"O"
              },
              {
                "id":15,
                "orth":"psiconor",
                "space":" ",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Gender=Fem|Number=Sing|VerbForm=Part",
                "lemma":"psiconor",
                "head":-1,
                "dep":"obj",
                "ner":"O"
              },
              {
                "id":16,
                "orth":"hace",
                "space":" ",
                "tag":"VERB",
                "pos":"VERB",
                "morph":"Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin",
                "lemma":"hacer",
                "head":-2,
                "dep":"advcl",
                "ner":"O"
              },
              {
                "id":17,
                "orth":"1",
                "space":" ",
                "tag":"NUM",
                "pos":"NUM",
                "morph":"NumForm=Digit|NumType=Card",
                "lemma":"1",
                "head":1,
                "dep":"nummod",
                "ner":"O"
              },
              {
                "id":18,
                "orth":"mes",
                "space":" ",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Masc|Number=Sing",
                "lemma":"mes",
                "head":-2,
                "dep":"obj",
                "ner":"O"
              },
              {
                "id":19,
                "orth":"en",
                "space":" ",
                "tag":"ADP",
                "pos":"ADP",
                "morph":"AdpType=Prep",
                "lemma":"en",
                "head":1,
                "dep":"case",
                "ner":"O"
              },
              {
                "id":20,
                "orth":"forma",
                "space":" ",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Fem|Number=Sing",
                "lemma":"formar",
                "head":-8,
                "dep":"obl",
                "ner":"O"
              },
              {
                "id":21,
                "orth":"progresiva",
                "space":"",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Gender=Fem|Number=Sing",
                "lemma":"progresivo",
                "head":-1,
                "dep":"amod",
                "ner":"O"
              },
              {
                "id":22,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"Gender=Masc|Number=Sing",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":23,
                "orth":"motivo",
                "space":" ",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Masc|Number=Sing",
                "lemma":"motivar",
                "head":-11,
                "dep":"obj",
                "ner":"B-Term"
              },
              {
                "id":24,
                "orth":"de",
                "space":" ",
                "tag":"ADP",
                "pos":"ADP",
                "morph":"AdpType=Prep",
                "lemma":"de",
                "head":1,
                "dep":"case",
                "ner":"I-Term"
              },
              {
                "id":25,
                "orth":"consulta",
                "space":"",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Fem|Number=Sing",
                "lemma":"consultar",
                "head":-13,
                "dep":"obl",
                "ner":"L-Term"
              },
              {
                "id":26,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":27,
                "orth":"rinofaringitis",
                "space":" ",
                "tag":"NOUN",
                "pos":"NOUN",
                "morph":"Gender=Masc|Number=Plur",
                "lemma":"rinofaringitis",
                "head":-15,
                "dep":"obj",
                "ner":"U-Term"
              },
              {
                "id":28,
                "orth":"aguda",
                "space":" ",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Gender=Fem|Number=Sing",
                "lemma":"agudo",
                "head":-1,
                "dep":"amod",
                "ner":"O"
              },
              {
                "id":29,
                "orth":"viral",
                "space":"",
                "tag":"ADJ",
                "pos":"ADJ",
                "morph":"Number=Sing",
                "lemma":"viral",
                "head":-2,
                "dep":"amod",
                "ner":"O"
              },
              {
                "id":30,
                "orth":"\n",
                "space":"",
                "tag":"SPACE",
                "pos":"SPACE",
                "morph":"",
                "lemma":"\n",
                "head":-1,
                "dep":"",
                "ner":"O"
              },
              {
                "id":31,
                "orth":"no",
                "space":" ",
                "tag":"ADV",
                "pos":"ADV",
                "morph":"Polarity=Neg",
                "lemma":"no",
                "head":1,
                "dep":"advmod",
                "ner":"O"
              },
              {
                "id":32,
                "orth":"fiebre",
                "space":"",
                "tag":"VERB",
                "pos":"VERB",
                "morph":"Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin",
                "lemma":"fiebre",
                "head":-20,
                "dep":"obj",
                "ner":"U-Term"
              }
            ],
            "brackets":[

            ]
          }
        ],
        "cats":[

        ],
        "entities":[
          [
            8,
            17,
            "Term"
          ],
          [
            18,
            39,
            "Term"
          ],
          [
            40,
            56,
            "Term"
          ],
          [
            121,
            139,
            "Term"
          ],
          [
            140,
            154,
            "Term"
          ],
          [
            170,
            176,
            "Term"
          ]
        ],
        "links":[

        ]
      }
    ]
  }
]`

Are all the components initialized with a proper set of labels? I'm not sure about this.

In the config I have frozen the tagger and parser components. Is there some initialization that I'm missing?

Thanks again.

svlandeg commented 4 years ago

I just noticed the frozen components. I think there's a bug related to those, that they are not disabled when calling the evaluation. I'll have a further look. Your stack trace is helpful, thanks :-)

svlandeg commented 4 years ago

Why do you have these frozen components? In general, you want frozen component for models that have already been trained and you don't want/need to update their weights, but you still need them in the pipeline at run-time (or evaluation-time). Imagine for instance training a named entity linker, but freezing the named entity recognizer that you trained before. So, those frozen components should already have been initialized and trained before you run this new training loop.

If you just wanted to train an NER, and not bother with the tagger and parser, you can create a config file for that by running init config ... -p "ner", or just remove the irrelevant sections from the config you have. But I'm not sure what your use-case is exactly.

fcggamou commented 4 years ago

Why do you have these frozen components? In general, you want frozen component for models that have already been trained and you don't want/need to update their weights, but you still need them in the pipeline at run-time (or evaluation-time). Imagine for instance training a named entity linker, but freezing the named entity recognizer that you trained before. So, those frozen components should already have been initialized and trained before you run this new training loop.

If you just wanted to train an NER, and not bother with the tagger and parser, you can create a config file for that by running init config ... -p "ner", or just remove the irrelevant sections from the config you have. But I'm not sure what your use-case is exactly.

Ahh! That was just a misunderstanding of mine. My use case is that I only need the NER indeed. I also need the transformer I assume, not sure if I should also include that in the -p argument?

Thanks a lot!

I will post an update once I am able to try this suggestion..

svlandeg commented 4 years ago

No, the "transformer" (or "tok2vec") comes "automatically" in the config, depending on the settings for efficiency/accuracy and gpu/cpu. If you'll just take the references to parser & tagger out of your config, I'm pretty sure it'll just work ;-)

fcggamou commented 4 years ago

No, the "transformer" (or "tok2vec") comes "automatically" in the config, depending on the settings for efficiency/accuracy and gpu/cpu. If you'll just take the references to parser & tagger out of your config, I'm pretty sure it'll just work ;-)

Great, that makes sense!

Indeed after your suggestion I am no longer getting that error message, but am getting a different one now though:

2020-10-09 12:05:12.258602: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
✔ Created output directory: model
Config overrides from CLI: ['paths.train', 'paths.dev']
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
Set up nlp object from config
Loading corpus from path: test.spacy
Loading corpus from path: train.spacy
Pipeline: ['transformer', 'ner']
Created vocabulary
Finished initializing nlp object
Downloading: 100% 524/524 [00:00<00:00, 403kB/s]
Downloading: 100% 981k/981k [00:00<00:00, 2.26MB/s]
Downloading: 100% 523k/523k [00:00<00:00, 1.23MB/s]
Downloading: 100% 150/150 [00:00<00:00, 107kB/s]
Downloading: 100% 2.00/2.00 [00:00<00:00, 1.47kB/s]
Downloading: 100% 507M/507M [00:06<00:00, 81.6MB/s]
Initialized pipeline components: ['transformer', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
Loading corpus from path: test.spacy
Loading corpus from path: train.spacy
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.0
E     #        LOSS TRANSFORMER   LOSS NER   ENTS_F   ENTS_P   ENTS_R   SCORE 
---   ------   ----------------   --------   ------   ------   ------   ------
⚠ Aborting and saving the final best model. Encountered exception:
'Transformer' object has no attribute 'finish_update'
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.6/dist-packages/spacy/cli/_util.py", line 65, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.6/dist-packages/spacy/cli/train.py", line 59, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 105, in train
    raise e
  File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 85, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/usr/local/lib/python3.6/dist-packages/spacy/training/loop.py", line 194, in train_while_improving
    proc.finish_update(optimizer)
AttributeError: 'Transformer' object has no attribute 'finish_update'

Any suggestion is welcome.

Thanks!

svlandeg commented 4 years ago

I think that issue should be fixed in the very latest updates, could you upgrade to spacy-nightly 3.0.0a36 and spacy-transformers 1.0.0a22? It has to do with the is_trainable setting and the inheritance from TrainablePipe, which was in flux the last few days.

(FYI - I find it easier to address one issue at the time, which also makes things more comprehensive if anyone stumbles upon an issue via google in the future. So don't hesitate to create a new issue when encountering a new (unrelated) problem)

fcggamou commented 4 years ago

Thank you Sofie, using those versions fixed the issue!

I will take your advice and create separate issues for separate problems from now on 👍

Thanks, Fede.

svlandeg commented 4 years ago

Awesome, happy to hear it works now!

github-actions[bot] commented 2 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.