Noun chucks returning empty list for the custom pipeline

[gunalanlakshmanan]

Discussed in https://github.com/explosion/spaCy/discussions/11059

^{Originally posted by **gunalanlakshmanan** June 30, 2022} I have trained a custom pipeline and with new ner and Spancat component and used the parser and tagger from source. Here is my config ``` [paths] train = null dev = null vectors = null init_tok2vec = null raw_text = null ner_labels = null spancat_labels = null [system] gpu_allocator = null seed = 0 [nlp] lang = "en" pipeline = ["tok2vec","tagger","parser","ner","spancat"] batch_size = 1000 disabled = [] before_creation = null after_creation = null after_pipeline_creation = null tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"} [components] [components.tagger] source = "en_core_web_sm" replace_listeners = ["model.tok2vec"] [components.parser] source = "en_core_web_sm" replace_listeners = ["model.tok2vec"] [components.ner] factory = "ner" incorrect_spans_key = null moves = null scorer = {"@scorers":"spacy.ner_scorer.v1"} update_with_oracle_cut_size = 100 [components.ner.model] @architectures = "spacy.TransitionBasedParser.v2" state_type = "ner" extra_state_tokens = false hidden_width = 64 maxout_pieces = 2 use_upper = true nO = null [components.ner.model.tok2vec] @architectures = "spacy.Tok2VecListener.v1" width = ${components.tok2vec.model.encode.width} upstream = "*" [components.spancat] factory = "spancat" max_positive = null scorer = {"@scorers":"spacy.spancat_scorer.v1"} spans_key = "sc" threshold = 0.5 [components.spancat.model] @architectures = "spacy.SpanCategorizer.v1" [components.spancat.model.reducer] @layers = "spacy.mean_max_reducer.v1" hidden_size = 128 [components.spancat.model.scorer] @layers = "spacy.LinearLogistic.v1" nO = null nI = null [components.spancat.model.tok2vec] @architectures = "spacy.Tok2VecListener.v1" width = ${components.tok2vec.model.encode.width} upstream = "*" [components.spancat.suggester] @misc = "spacy.ngram_suggester.v1" sizes = [1, 2] [components.tok2vec] factory = "tok2vec" [components.tok2vec.model] @architectures = "spacy.Tok2Vec.v2" [components.tok2vec.model.embed] @architectures = "spacy.MultiHashEmbed.v2" width = ${components.tok2vec.model.encode.width} attrs = ["ORTH","SHAPE"] rows = [5000,2500] include_static_vectors = false [components.tok2vec.model.encode] @architectures = "spacy.MaxoutWindowEncoder.v2" width = 96 depth = 4 window_size = 1 maxout_pieces = 3 [corpora] [corpora.dev] @readers = "spacy.Corpus.v1" path = ${paths.dev} max_length = 0 gold_preproc = false limit = 0 augmenter = null [corpora.pretrain] @readers = "spacy.JsonlCorpus.v1" path = ${paths.raw_text} min_length = 5 max_length = 500 limit = 0 [corpora.train] @readers = "spacy.Corpus.v1" path = ${paths.train} max_length = 0 gold_preproc = false limit = 0 augmenter = null [training] dev_corpus = "corpora.dev" train_corpus = "corpora.train" seed = ${system.seed} gpu_allocator = ${system.gpu_allocator} dropout = 0.1 accumulate_gradient = 1 patience = 1600 max_epochs = 100 max_steps = 20000 eval_frequency = 200 frozen_components = ["tok2vec", "tagger", "parser"] annotating_components = ["tok2vec"] before_to_disk = null [training.batcher] @batchers = "spacy.batch_by_words.v1" discard_oversize = false tolerance = 0.2 get_length = null [training.batcher.size] @schedules = "compounding.v1" start = 100 stop = 1000 compound = 1.001 t = 0.0 [training.logger] @loggers = "spacy.ConsoleLogger.v1" progress_bar = false [training.optimizer] @optimizers = "Adam.v1" beta1 = 0.9 beta2 = 0.999 L2_is_weight_decay = true L2 = 0.01 grad_clip = 1.0 use_averages = false eps = 0.00000001 learn_rate = 0.001 [training.score_weights] tag_acc = 0.25 dep_uas = 0.12 dep_las = 0.12 dep_las_per_type = null sents_p = null sents_r = null sents_f = 0.0 ents_f = 0.25 ents_p = 0.0 ents_r = 0.0 ents_per_type = null spans_sc_f = 0.25 spans_sc_p = 0.0 spans_sc_r = 0.0 [pretraining] max_epochs = 1500 dropout = 0.2 n_save_every = null n_save_epoch = null component = "tok2vec" layer = "" corpus = "corpora.pretrain" [pretraining.batcher] @batchers = "spacy.batch_by_words.v1" size = 3000 discard_oversize = false tolerance = 0.2 get_length = null [pretraining.objective] @architectures = "spacy.PretrainCharacters.v1" maxout_pieces = 3 hidden_size = 300 n_characters = 4 [pretraining.optimizer] @optimizers = "Adam.v1" beta1 = 0.9 beta2 = 0.999 L2_is_weight_decay = true L2 = 0.01 grad_clip = 1.0 use_averages = true eps = 0.00000001 learn_rate = 0.001 [initialize] vectors = ${paths.vectors} init_tok2vec = ${paths.init_tok2vec} vocab_data = null lookups = null after_init = null [initialize.before_init] @callbacks = "sms_tokenizer" [initialize.components] [initialize.components.ner] [initialize.components.ner.labels] @readers = "spacy.read_labels.v1" path = ${paths.ner_labels} [initialize.components.spancat] [initialize.components.spancat.labels] @readers = "spacy.read_labels.v1" path = ${paths.spancat_labels} require = false [initialize.tokenizer] ``` I have checked the syntax iterators in Default, it contains noun chucks. The default NLP model "en_core_web_sm" provides noun chunks for same sentence. ``` default_doc = nlp("It's a beautiful dog") list(default_doc.noun_chunks) Out[17]: [It, a beautiful dog] ``` ``` custom_doc = custom_nlp("It's a beautiful dog") list(custom_doc.noun_chunks) Out[15]: [] ``` The parser output is same for both default and custom model. ``` for token in default_doc: print(token.text, token.tag_) It PRP 's VBZ a DT beautiful JJ dog NN ``` ``` for token in custom_doc: print(token.text, token.tag_) It PRP 's VBZ a DT beautiful JJ dog NN ``` I am not sure what I am missing here. I am not getting the noun chunks and pos_ from the doc created using the custom model. Can you please help me on finding the problem here? Thanks in advance

[polm]

Closing as duplicate of #11059. Please don't open a Discussion and an Issue about the same thing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.