search

BrikerMan / Kashgari

Kashgari is a production-level NLP Transfer learning framework built on top of tf.keras for text-labeling and text-classification, includes Word2Vec, BERT, and GPT2 Language Embedding.
http://kashgari.readthedocs.io/
Apache License 2.0
2.39k stars 441 forks source link

[Question] predict only returning 9 outputs for labels #164

Closed MattHuntebrinker closed 5 years ago

MattHuntebrinker commented 5 years ago

Regardless of my input len, and only when running predict, not eval, the output argmax will only return 9 labels.

It changes for each input sentence, but it will only return a maximum of 9 predictions.

['A', 'purple ', 'pig ', 'and:', 'a', 'green', 'donkey', 'flew', 'a', 'kite', 'in', 'the', 'middle', 'of', 'the', 'night', 'and', 'ended', 'up', 'sunburnt']
[[[1.00000000e+00 1.16705285e-08 3.15263260e-10 3.71873560e-10
   1.27021049e-09 8.63430855e-13 9.55439720e-12 1.98695465e-12
   1.61817044e-12 8.36667568e-14 1.03332409e-09 6.22354772e-12
   2.07860534e-11 8.08257676e-14]
  [3.22285615e-07 9.99992013e-01 2.65579024e-06 7.89299804e-07
   4.51470271e-07 9.18732496e-07 3.28745102e-08 3.05571994e-08
   1.33486890e-06 2.44629447e-12 1.30242540e-06 6.86431108e-08
   3.43951617e-10 2.07822888e-14]
  [3.93861388e-10 9.99984384e-01 5.03810770e-06 1.59945512e-06
   1.19765321e-06 3.18203115e-06 1.54177854e-07 6.84599530e-11
   3.08716153e-06 6.54515884e-13 1.26170050e-06 7.68855726e-12
   6.54374048e-08 9.38123972e-15]
  [1.27381233e-13 9.99997377e-01 9.08361244e-07 7.93735751e-07
   4.61653968e-07 1.68776253e-07 1.12256493e-09 2.85287778e-14
   1.06929235e-07 1.65791866e-15 6.28999146e-08 3.26833148e-19
   2.54236543e-08 1.20595045e-16]
  [5.25962780e-12 9.99994040e-01 1.09953908e-06 1.52247640e-06
   7.89339765e-07 5.36937137e-07 8.76613093e-09 2.64227659e-12
   6.01736303e-07 3.29582607e-12 1.09808980e-06 3.14427440e-09
   2.01072027e-07 3.80130531e-14]
  [3.74625608e-10 1.47379615e-04 1.43367984e-07 1.93348171e-07
   1.79619342e-08 6.95794643e-06 2.11896563e-08 1.50931501e-09
   1.22247193e-05 1.00118143e-10 3.95079296e-05 9.99793470e-01
   5.97124838e-08 2.66669824e-13]
  [2.81098007e-08 2.55360007e-01 8.95073346e-04 9.12376388e-04
   5.65130504e-05 4.76730764e-02 1.99197129e-05 2.19619764e-07
   4.18915749e-02 1.54858071e-10 6.53187156e-01 2.06247091e-06
   2.04470416e-06 2.22696271e-11]
  [3.25567306e-08 6.01284146e-01 5.30494004e-02 6.82911500e-02
   7.79066933e-03 3.75758335e-02 7.49568644e-05 2.32547677e-07
   1.70825701e-02 9.80864973e-11 2.14847207e-01 1.58744268e-11
   3.84300483e-06 4.02588206e-12]
  [6.58055166e-09 7.82069921e-01 6.49503022e-02 8.31069499e-02
   9.71517432e-03 2.07212083e-02 4.90007165e-04 1.45882223e-06
   1.54230986e-02 2.20132551e-10 2.34999638e-02 5.21106316e-15
   2.18832847e-05 4.03389697e-13]
  [6.54785959e-09 5.84027350e-01 6.81363791e-02 8.64769071e-02
   1.13065075e-02 1.01214282e-01 3.53630516e-03 3.72529917e-06
   1.13552153e-01 1.85934496e-07 1.00733181e-02 4.27359641e-11
   2.16728877e-02 1.14726050e-11]]]
[['O', 'O', 'O', 'O', 'B-SS', 'I-SS', 'O', 'O', 'O']]

It doesn't matter how the sentence changes, the labels will be correct, but only spit out 9 of them. I wasn't sure if this is a bug or a question, but if you need more information please let me know.

BrikerMan commented 5 years ago

I need more details, please run model.tf_model.summary() to print out model details, and which function you use to predict?

MattHuntebrinker commented 5 years ago

here's the code used to train and predict, since i'm not exactly sure what you're looking for. 

from kashgari.corpus import CONLL2003ENCorpus
import kashgari
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = CONLL2003ENCorpus.load_data('train')
valid_x, valid_y = CONLL2003ENCorpus.load_data('valid')
test_x, test_y = CONLL2003ENCorpus.load_data('test')

bert_embed = BERTEmbedding('bert',
                           task=kashgari.LABELING,
                           sequence_length=100)

model = BiLSTM_Model(bert_embed)
model.fit(train_x, train_y, valid_x, valid_y)

model.evaluate(test_x, test_y)

model.save('saved_ner_model')

x = ['A', 'purple ', 'pig ', 'and:', 'a', 'green', 'donkey', 'flew', 'a', 'kite', 'in', 'the', 'middle', 'of', 'the', 'night', 'and', 'ended', 'up', 'sunburnt']

loaded_model = kashgari.utils.load_model('saved_ner_model')
loaded_model.predict([x])
BrikerMan commented 5 years ago

I need the result of this code

import logging
logging.basicConfig(level='DEBUG')
print(kashgari.__version__)

loaded_model.tf_model.summary()
loaded_model.predict([x], debug_info=True)
MattHuntebrinker commented 5 years ago 
0.5.0
x[0]: ['Supplement', 'Facts', 'Serving', 'Size:', '1tsp', '5mL', 'Servings', 'Per', 'Container:', '19', 'Calories', '11', 'Cal', 'SUGARS†', '3', 'g', 'Total', 'Carbohydrate**', '3', 'g', '1%', 'SENNA', 'POD', 'CONCENTRATE†', '167', 'mg', 'PURIFIED', 'WATER,', 'SUCROSE,', 'MALTODEXTRIN,', 'GLYCERIN,', 'SODIUM', 'BENZOATE', '(PRESERVATIVE),', 'ARTIFICIAL', 'FLAVOR,', 'METHYLPARABEN', '(PRESERVATIVE),', 'CITRIC', 'ACID,', 'PROPYLPARABEN', '(PRESERVATIVE).', 'percent', 'daily', 'value', 'based', 'on', '2000', 'calorie', 'diet']
y[0]: ['O', 'O', 'O', 'O', 'B-SS', 'I-SS', 'O', 'O', 'O', 'B-SC', 'B-N', 'B-A', 'I-A', 'B-N', 'B-A', 'I-A', 'B-N', 'I-N', 'B-A', 'I-A', 'B-DV', 'B-I', 'I-I', 'I-I', 'B-A', 'I-A', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']
DEBUG:root:loaded 3194 samples from ./data/valid.txt. Sample:
x[0]: ['Supplement', 'Facts', 'Serving', 'Size:', '1', 'SCOOP', '8.5g', 'Servings', 'Per', 'Container:', '10', 'Calories', '15Cal', 'Sodium', '510mg', '22%', 'GOBHB', 'BETA', 'HYDROXYBUTYRATE**', '6g', 'Potassium', '200mg', '4%', 'Calcium', '260mg', '20%', 'Magnesium', '75mg', '18%', 'SODIUM', 'BHB,', 'CALCIUM', 'BHB,', 'MAGNESIUM', 'BHB,', 'POTASSIUM', 'CITRATE,', 'CITRIC', 'ACID,', 'MALIC', 'ACID,', 'NATURAL', 'FLAVORS,', 'SILICON', 'DIOXIDE,', 'STEVIA', 'EXTRACT,', 'BETA', 'CAROTENE', '(NO', 'COLORS', 'FROM', 'ARTIFICIAL', 'SOURCES)']
y[0]: ['O', 'O', 'O', 'O', 'B-SS', 'I-SS', 'I-SS', 'O', 'O', 'O', 'B-SC', 'B-N', 'B-A', 'B-N', 'B-A', 'B-DV', 'B-I', 'I-I', 'I-I', 'B-A', 'B-N', 'B-A', 'B-DV', 'B-N', 'B-A', 'B-DV', 'B-N', 'B-A', 'B-DV', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']
DEBUG:root:loaded 3074 samples from ./data/test.txt. Sample:
x[0]: ['Supplement', 'Facts', 'Serving', 'Size:', 'SOFTGEL', '1Gel', 'VITAMIN', 'D3', '(AS', 'CHOLECALCIFEROL)', '(1000', 'IU)', '25', 'mcg', '125%', 'SOYBEAN', 'OIL,', 'GELATIN,', 'GLYCERIN,', 'WATER.', 'percent', 'daily', 'value', 'based', 'on', '2000', 'calorie', 'diet']
y[0]: ['O', 'O', 'O', 'O', 'B-SS', 'I-SS', 'B-N', 'I-N', 'I-N', 'I-N', 'I-N', 'I-N', 'B-A', 'I-A', 'B-DV', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']
WARNING:root:Sequence length will auto set at 95% of sequence length
DEBUG:root:need to build after build_word2idx
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input-Token (InputLayer)        (None, 10)           0                                            
__________________________________________________________________________________________________
Input-Segment (InputLayer)      (None, 10)           0                                            
__________________________________________________________________________________________________
Embedding-Token (TokenEmbedding [(None, 10, 1024), ( 31254528    Input-Token[0][0]                
__________________________________________________________________________________________________
Embedding-Segment (Embedding)   (None, 10, 1024)     2048        Input-Segment[0][0]              
__________________________________________________________________________________________________
Embedding-Token-Segment (Add)   (None, 10, 1024)     0           Embedding-Token[0][0]            
                                                                 Embedding-Segment[0][0]          
__________________________________________________________________________________________________
Embedding-Position (PositionEmb (None, 10, 1024)     10240       Embedding-Token-Segment[0][0]    
__________________________________________________________________________________________________
Embedding-Dropout (Dropout)     (None, 10, 1024)     0           Embedding-Position[0][0]         
__________________________________________________________________________________________________
Embedding-Norm (LayerNormalizat (None, 10, 1024)     2048        Embedding-Dropout[0][0]          
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Embedding-Norm[0][0]             
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-1-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 10, 1024)     0           Embedding-Norm[0][0]             
                                                                 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-1-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-1-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-1-MultiHeadSelfAttention-
                                                                 Encoder-1-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-1-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-1-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-1-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-2-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-1-FeedForward-Norm[0][0] 
                                                                 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-2-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-2-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-2-MultiHeadSelfAttention-
                                                                 Encoder-2-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-2-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-2-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-2-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-3-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-2-FeedForward-Norm[0][0] 
                                                                 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-3-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-3-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-3-MultiHeadSelfAttention-
                                                                 Encoder-3-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-3-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-3-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-3-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-4-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-3-FeedForward-Norm[0][0] 
                                                                 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-4-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-4-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-4-MultiHeadSelfAttention-
                                                                 Encoder-4-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-4-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-4-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-4-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-5-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-4-FeedForward-Norm[0][0] 
                                                                 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-5-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-5-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-5-MultiHeadSelfAttention-
                                                                 Encoder-5-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-5-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-5-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-5-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-6-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-5-FeedForward-Norm[0][0] 
                                                                 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-6-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-6-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-6-MultiHeadSelfAttention-
                                                                 Encoder-6-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-6-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-6-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-6-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-7-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-6-FeedForward-Norm[0][0] 
                                                                 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-7-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-7-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-7-MultiHeadSelfAttention-
                                                                 Encoder-7-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-7-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-7-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-7-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-8-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-7-FeedForward-Norm[0][0] 
                                                                 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-8-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-8-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-8-MultiHeadSelfAttention-
                                                                 Encoder-8-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-8-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-8-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 10, 1024)     4198400     Encoder-8-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-9-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 10, 1024)     0           Encoder-8-FeedForward-Norm[0][0] 
                                                                 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 10, 1024)     2048        Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward (FeedForw (None, 10, 1024)     8393728     Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward-Dropout ( (None, 10, 1024)     0           Encoder-9-FeedForward[0][0]      
__________________________________________________________________________________________________
Encoder-9-FeedForward-Add (Add) (None, 10, 1024)     0           Encoder-9-MultiHeadSelfAttention-
                                                                 Encoder-9-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-9-FeedForward-Norm (Lay (None, 10, 1024)     2048        Encoder-9-FeedForward-Add[0][0]  
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-9-FeedForward-Norm[0][0] 
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-9-FeedForward-Norm[0][0] 
                                                                 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-10-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-10-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-10-MultiHeadSelfAttention
                                                                 Encoder-10-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-10-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-10-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-10-FeedForward-Norm[0][0]
                                                                 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-11-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-11-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-11-MultiHeadSelfAttention
                                                                 Encoder-11-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-11-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-11-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-11-FeedForward-Norm[0][0]
                                                                 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-12-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-12-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-12-MultiHeadSelfAttention
                                                                 Encoder-12-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-12-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-12-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-12-FeedForward-Norm[0][0]
                                                                 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-13-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-13-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-13-MultiHeadSelfAttention
                                                                 Encoder-13-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-13-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-13-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-13-FeedForward-Norm[0][0]
                                                                 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-14-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-14-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-14-MultiHeadSelfAttention
                                                                 Encoder-14-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-14-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-14-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-14-FeedForward-Norm[0][0]
                                                                 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-15-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-15-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-15-MultiHeadSelfAttention
                                                                 Encoder-15-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-15-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-15-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-15-FeedForward-Norm[0][0]
                                                                 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-16-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-16-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-16-MultiHeadSelfAttention
                                                                 Encoder-16-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-16-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-16-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-16-FeedForward-Norm[0][0]
                                                                 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-17-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-17-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-17-MultiHeadSelfAttention
                                                                 Encoder-17-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-17-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-17-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-17-FeedForward-Norm[0][0]
                                                                 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-18-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-18-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-18-MultiHeadSelfAttention
                                                                 Encoder-18-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-18-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-18-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-18-FeedForward-Norm[0][0]
                                                                 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-19-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-19-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-19-MultiHeadSelfAttention
                                                                 Encoder-19-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-19-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-19-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-19-FeedForward-Norm[0][0]
                                                                 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-20-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-20-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-20-MultiHeadSelfAttention
                                                                 Encoder-20-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-20-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-20-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-20-FeedForward-Norm[0][0]
                                                                 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-21-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-21-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-21-MultiHeadSelfAttention
                                                                 Encoder-21-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-21-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-21-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-21-FeedForward-Norm[0][0]
                                                                 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-22-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-22-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-22-MultiHeadSelfAttention
                                                                 Encoder-22-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-22-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-22-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-22-FeedForward-Norm[0][0]
                                                                 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-23-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-23-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-23-MultiHeadSelfAttention
                                                                 Encoder-23-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-23-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 10, 1024)     4198400     Encoder-23-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 10, 1024)     0           Encoder-23-FeedForward-Norm[0][0]
                                                                 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 10, 1024)     2048        Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward (FeedFor (None, 10, 1024)     8393728     Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward-Dropout  (None, 10, 1024)     0           Encoder-24-FeedForward[0][0]     
__________________________________________________________________________________________________
Encoder-24-FeedForward-Add (Add (None, 10, 1024)     0           Encoder-24-MultiHeadSelfAttention
                                                                 Encoder-24-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Norm (La (None, 10, 1024)     2048        Encoder-24-FeedForward-Add[0][0] 
__________________________________________________________________________________________________
Encoder-Output (Concatenate)    (None, 10, 4096)     0           Encoder-21-FeedForward-Norm[0][0]
                                                                 Encoder-22-FeedForward-Norm[0][0]
                                                                 Encoder-23-FeedForward-Norm[0][0]
                                                                 Encoder-24-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
non_masking_layer (NonMaskingLa (None, 10, 4096)     0           Encoder-Output[0][0]             
__________________________________________________________________________________________________
layer_blstm (Bidirectional)     (None, 10, 256)      4326400     non_masking_layer[0][0]          
__________________________________________________________________________________________________
layer_dropout (Dropout)         (None, 10, 256)      0           layer_blstm[0][0]                
__________________________________________________________________________________________________
layer_time_distributed (TimeDis (None, 10, 14)       3598        layer_dropout[0][0]              
__________________________________________________________________________________________________
activation (Activation)         (None, 10, 14)       0           layer_time_distributed[0][0]     
==================================================================================================
Total params: 337,908,238
Trainable params: 4,329,998
Non-trainable params: 333,578,240
__________________________________________________________________________________________________
[['O', 'I-SS', 'O', 'O', 'I-SS', 'I-SS', 'I-SS', 'O', 'O']]
BrikerMan commented 5 years ago

It seems the sequence_length=100 not working, and sequence_length set to 10 since most of your data length shorter than 10. Since BERT will add a special token [CLS] at the start of the sequence, you get result with a length of 9.

Please run this code below and help me figure out is that an issue related to the save and load process.

import logging
logging.basicConfig(level='DEBUG')

from kashgari.corpus import CONLL2003ENCorpus
import kashgari
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = CONLL2003ENCorpus.load_data('train')
valid_x, valid_y = CONLL2003ENCorpus.load_data('valid')
test_x, test_y = CONLL2003ENCorpus.load_data('test')

bert_embed = BERTEmbedding('bert',
                           task=kashgari.LABELING,
                           sequence_length=100)

model = BiLSTM_Model(bert_embed)
model.build_model(train_x, train_y, valid_x, valid_y)
MattHuntebrinker commented 5 years ago

I get the exact output as I did in the last comment when running build model

BrikerMan commented 5 years ago

I get the exact output as I did in the last comment when running build model

I have tried this code on colab, unable to reproduce the issue, here is the colab link. Maybe update to the 0.5.1 version then try again.