search

explosion / spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python
https://spacy.io
MIT License
29.7k stars 4.36k forks source link

NER training with new entity array shape error #3189

Closed nyejon closed 5 years ago

nyejon commented 5 years ago

How to reproduce the behaviour

Run the standard new entity training example on https://spacy.io/usage/training#section-ner

Your Environment

Info about spaCy

I get a "ValueError: could not broadcast input array from shape (77,64) into shape (73,64)"

I have tried both the english md and lg models iwth the same results

ines commented 5 years ago

Could you check the output of python -m spacy validate to make sure you're using the correct compatible models (especially since you're on spacy-nightly)?

nyejon commented 5 years ago

===================== Installed models (spaCy v2.1.0a6) ===================== ℹ spaCy installation: /.local/share/virtualenvs/machine-learning-mJ7mbo3c/lib/python3.7/site-packages/spacy

TYPE NAME MODEL VERSION package en-core-web-md en_core_web_md 2.1.0a5 ✔ package en-core-web-lg en_core_web_lg 2.1.0a5 ✔ package de-core-news-md de_core_news_md 2.1.0a5 ✔

nyejon commented 5 years ago

Hi, to extend this issue - If I change the ner config after training I get:

{ "beam_width":1, "beam_density":0.0, "cnn_maxout_pieces":3, "deprecation_fixes":{ "vectors_name":"en_model.vectors" }, "nr_class":89, <----- This was incorrectly written out as 73 "hidden_depth":1, "token_vector_width":96, "hidden_width":64, "maxout_pieces":2, "pretrained_vectors":"en_model.vectors", "bilstm_depth":0 }

If I change the value of NR class to 89 as the new model expects it loads but everything is labelled as a NORP

ines commented 5 years ago

@nyejon Are you using the latest nightly, 2.1.0a7? We definitely fixed a bug that might have caused problems like "everything is labelled XY", so if you're still using a6, try again with the new nightly and see if the problem is resolved.

nyejon commented 5 years ago

Yes I am. This was tested again because I had upgraded to see if it works now. The NER config file seems to also get incorrectly created, so the model does not load unless that value is changed manually.

honnibal commented 5 years ago

Found the bug, and fixed a number of other issues associated with adding labels to a pre-trained model. It's definitely a hard workflow to support correctly, but we're really committed to making it work, as it's super valuable.

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.