search

lavis-nlp / jerex

PyTorch code for JEREX: Joint Entity-Level Relation Extractor
MIT License
61 stars 15 forks source link

performance metrics is zero #10

Closed injala-kumar closed 2 years ago

injala-kumar commented 2 years ago

Can you please help me with the following things. I have create a custom dataset with same structure as docred and tried to train the model. First of all it's not seems training by looking at performace metrics. Also after some epochs it's gives error and stop training. you can check the following logs. 

datasets:
  train_path: ./data/datasets/docred_joint/train_joint.json
  valid_path: ./data/datasets/docred_joint/dev_joint.json
  test_path: null
  types_path: ./data/datasets/docred_joint/types.json
model:
  model_type: joint_multi_instance
  encoder_path: bert-base-cased
  tokenizer_path: bert-base-cased
  mention_threshold: 0.85
  coref_threshold: 0.85
  rel_threshold: 0.6
  prop_drop: 0.1
  meta_embedding_size: 25
  size_embeddings_count: 40
  ed_embeddings_count: 300
  token_dist_embeddings_count: 700
  sentence_dist_embeddings_count: 50
  position_embeddings_count: 700
sampling:
  neg_mention_count: 200
  neg_coref_count: 200
  neg_relation_count: 200
  max_span_size: 10
  sampling_processes: 8
  neg_mention_overlap_ratio: 0.5
  lowercase: false
loss:
  mention_weight: 1.0
  coref_weight: 1.0
  entity_weight: 0.25
  relation_weight: 1.0
inference:
  valid_batch_size: 1
  test_batch_size: 1
  max_spans: null
  max_coref_pairs: null
  max_rel_pairs: null
training:
  batch_size: 1
  min_epochs: 20
  max_epochs: 20
  lr: 5.0e-05
  lr_warmup: 0.1
  weight_decay: 0.01
  max_grad_norm: 1.0
  accumulate_grad_batches: 1
  max_spans: null
  max_coref_pairs: null
  max_rel_pairs: null
distribution:
  gpus: []
  accelerator: ''
  prepare_data_per_node: false
misc:
  store_predictions: true
  store_examples: true
  flush_logs_every_n_steps: 1000
  log_every_n_steps: 1000
  deterministic: false
  seed: null
  cache_path: null
  precision: 32
  profiler: null
  final_valid_evaluate: true

Parse dataset '/content/jerex/data/datasets/docred_joint/train_joint.json': 100% 260/260 [00:02<00:00, 111.80it/s]
Parse dataset '/content/jerex/data/datasets/docred_joint/dev_joint.json': 100% 30/30 [00:00<00:00, 120.30it/s]
Some weights of the model checkpoint at bert-base-cased were not used when initializing JointMultiInstanceModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'bert.pooler.dense.weight', 'bert.pooler.dense.bias']
- This IS expected if you are initializing JointMultiInstanceModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing JointMultiInstanceModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of JointMultiInstanceModel were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['mention_localization.linear.weight', 'mention_localization.linear.bias', 'mention_localization.mention_classifier.weight', 'mention_localization.mention_classifier.bias', 'mention_localization.size_embeddings.weight', 'coreference_resolution.coref_linear.weight', 'coreference_resolution.coref_linear.bias', 'coreference_resolution.coref_classifier.weight', 'coreference_resolution.coref_classifier.bias', 'coreference_resolution.coref_ed_embeddings.weight', 'entity_classification.linear.weight', 'entity_classification.linear.bias', 'entity_classification.entity_classifier.weight', 'entity_classification.entity_classifier.bias', 'relation_classification.pair_linear.weight', 'relation_classification.pair_linear.bias', 'relation_classification.rel_linear.weight', 'relation_classification.rel_linear.bias', 'relation_classification.rel_classifier.weight', 'relation_classification.rel_classifier.bias', 'relation_classification.token_distance_embeddings.weight', 'relation_classification.sentence_distance_embeddings.weight', 'relation_classification.entity_type_embeddings.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
GPU available: True, used: False
TPU available: False, using: 0 TPU cores
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:52: UserWarning: GPU available but not used. Set the --gpus flag when calling the script.
  warnings.warn(*args, **kwargs)

  | Name  | Type                    | Params
--------------------------------------------------
0 | model | JointMultiInstanceModel | 113 M 
--------------------------------------------------
113 M     Trainable params
0         Non-trainable params
113 M     Total params
455.678   Total estimated model params size (MB)
Epoch 0:  90% 260/290 [05:11<00:35,  1.20s/it, loss=0.568, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)

Epoch 0: 100% 290/290 [05:20<00:00,  1.10s/it, loss=0.568, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.51it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 0: 100% 290/290 [05:24<00:00,  1.12s/it, loss=0.568, v_num=0_0]
                                               tcmalloc: large alloc 1297850368 bytes == 0x56535c070000 @  0x7fa2102d4615 0x56526cb7b4cc 0x56526cc5b47a 0x56526cb81f0c 0x7fa20ae659e4 0x7fa20ae6e0e4 0x7fa20ae42bf0 0x7fa168968935 0x7fa16896426e 0x7fa16896940a 0x7fa20ae4313e 0x7fa20a8d8f98 0x56526cb7f098 0x56526cbf24d9 0x56526cbecced 0x56526cb7fbda 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbedc0d 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbedc0d 0x56526cb7fafa
tcmalloc: large alloc 1622319104 bytes == 0x5653a962a000 @  0x7fa2102d4615 0x56526cb7b4cc 0x56526cc5b47a 0x56526cb81f0c 0x7fa20ae659e4 0x7fa20ae6e0e4 0x7fa20ae42bf0 0x7fa168968935 0x7fa16896426e 0x7fa16896940a 0x7fa20ae4313e 0x7fa20a8d8f98 0x56526cb7f098 0x56526cbf24d9 0x56526cbecced 0x56526cb7fbda 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbedc0d 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbedc0d 0x56526cb7fafa
Epoch 1:  90% 260/290 [05:23<00:37,  1.24s/it, loss=0.817, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 1: 100% 290/290 [05:31<00:00,  1.14s/it, loss=0.817, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.49it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 1: 100% 290/290 [05:35<00:00,  1.16s/it, loss=0.817, v_num=0_0]
Epoch 2:  90% 260/290 [05:40<00:39,  1.31s/it, loss=0.343, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 2: 100% 290/290 [05:48<00:00,  1.20s/it, loss=0.343, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.51it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 2: 100% 290/290 [05:52<00:00,  1.22s/it, loss=0.343, v_num=0_0]
Epoch 3:  90% 260/290 [05:49<00:40,  1.34s/it, loss=0.175, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 3: 100% 290/290 [05:57<00:00,  1.23s/it, loss=0.175, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.50it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 3: 100% 290/290 [06:01<00:00,  1.25s/it, loss=0.175, v_num=0_0]
Epoch 4:  90% 260/290 [06:16<00:43,  1.45s/it, loss=0.12, v_num=0_0] 
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 4: 100% 290/290 [06:25<00:00,  1.33s/it, loss=0.12, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.39it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 4: 100% 290/290 [06:29<00:00,  1.34s/it, loss=0.12, v_num=0_0]
Epoch 5:  90% 260/290 [06:05<00:42,  1.41s/it, loss=0.117, v_num=0_0] 
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 5: 100% 290/290 [06:14<00:00,  1.29s/it, loss=0.117, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.52it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 5: 100% 290/290 [06:17<00:00,  1.30s/it, loss=0.117, v_num=0_0]
Epoch 6:  90% 260/290 [06:07<00:42,  1.41s/it, loss=0.1, v_num=0_0]  
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 6: 100% 290/290 [06:16<00:00,  1.30s/it, loss=0.1, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.47it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 6: 100% 290/290 [06:20<00:00,  1.31s/it, loss=0.1, v_num=0_0]
Epoch 7:  90% 260/290 [06:04<00:42,  1.40s/it, loss=0.153, v_num=0_0] 
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 7: 100% 290/290 [06:12<00:00,  1.29s/it, loss=0.153, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.49it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 7: 100% 290/290 [06:16<00:00,  1.30s/it, loss=0.153, v_num=0_0]
Epoch 8:  90% 260/290 [06:11<00:42,  1.43s/it, loss=0.0601, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 8: 100% 290/290 [06:20<00:00,  1.31s/it, loss=0.0601, v_num=0_0]
Validating: 100% 30/30 [00:13<00:00,  2.27it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 8: 100% 290/290 [06:25<00:00,  1.33s/it, loss=0.0601, v_num=0_0]
Epoch 9:  90% 260/290 [06:21<00:44,  1.47s/it, loss=0.0318, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 9: 100% 290/290 [06:30<00:00,  1.35s/it, loss=0.0318, v_num=0_0]
Validating: 100% 30/30 [00:13<00:00,  2.25it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 9: 100% 290/290 [06:35<00:00,  1.36s/it, loss=0.0318, v_num=0_0]
Epoch 10:  90% 260/290 [06:23<00:44,  1.47s/it, loss=0.0143, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 10: 100% 290/290 [06:33<00:00,  1.36s/it, loss=0.0143, v_num=0_0]
Validating: 100% 30/30 [00:14<00:00,  2.15it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 10: 100% 290/290 [06:37<00:00,  1.37s/it, loss=0.0143, v_num=0_0]
Epoch 11:  90% 260/290 [06:21<00:44,  1.47s/it, loss=0.0324, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Epoch 11:  97% 280/290 [06:24<00:13,  1.37s/it, loss=0.0324, v_num=0_0]
Traceback (most recent call last):
  File "./jerex_train.py", line 20, in train
    model.train(cfg)
  File "/content/jerex/jerex/model.py", line 341, in train
    trainer.fit(model, datamodule=data_module)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train
    self.train_loop.run_training_epoch()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 577, in run_training_epoch
    self.trainer.run_evaluation(on_epoch=True)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 725, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 166, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 177, in validation_step
    return self.training_type_plugin.validation_step(*args)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 131, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/content/jerex/jerex/model.py", line 126, in validation_step
    return self._inference(batch, batch_idx)
  File "/content/jerex/jerex/model.py", line 176, in _inference
    output = self(**batch, inference=True)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/jerex/jerex/model.py", line 106, in forward
    max_rel_pairs=max_rel_pairs, inference=inference)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/jerex/jerex/models/joint_models.py", line 144, in forward
    return self._forward_inference(*args, **kwargs)
  File "/content/jerex/jerex/models/joint_models.py", line 209, in _forward_inference
    mention_sample_masks, max_spans=max_spans, max_coref_pairs=max_coref_pairs)
  File "/content/jerex/jerex/models/joint_models.py", line 100, in _forward_inference_common
    valid_mentions, self._coref_threshold)
  File "/content/jerex/jerex/models/misc.py", line 135, in create_clusters
    assignments = agg_clustering.fit_predict(distances)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/cluster/_agglomerative.py", line 902, in fit_predict
    return super().fit_predict(X, y)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/base.py", line 462, in fit_predict
    self.fit(X)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/cluster/_agglomerative.py", line 796, in fit
    X = check_array(X, ensure_min_samples=2, estimator=self)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py", line 586, in check_array
    context))
ValueError: Found array with 1 sample(s) (shape=(1, 1)) while a minimum of 2 is required by AgglomerativeClustering.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
markus-eberts commented 2 years ago

Hi,

first of all, it seems as if each entity cluster in your dataset only consists of a single mention. Are you sure JEREX is the right approach here? Maybe a model that extracts relations between single mentions and not clusters of mentions is better suited in this case, like SpERT (https://github.com/lavis-nlp/spert -> self-promotion :) ). Maybe the code is missing a corner case handling in case there are no (positive) coreference samples. Unfortunately, I won't be able to look at the problem for two weeks since I'm on vacation as of tomorrow.

Does the model train when you only localize entities (by setting 'model_type' in configuration file to 'mention_localization')? Also, if you want to, you can send me the dataset by email (markus_eberts@yahoo.de). This will certainly help for debugging.

markus-eberts commented 2 years ago

I just added some corner case handling (412d3a94f9e866923da35799fc5c2f91ea7dc18e) before going on vacation. Please check if this already fixes your problem. Also, please try to increase the number of epochs (e.g. by setting training: min_epochs: 100 and training: max_epochs: 100 in configs/docred_joint/train.yaml) and wait till the training finishes.

markus-eberts commented 2 years ago

Please leave a comment if your issue is still not resolved.