Thank you a lot for your contribution! We tried to launch your project on KnowledgeNet Dataset (this version: https://github.com/schmidek/dygiepp/tree/multitask/dygie) and encounter with the following issue: the training does not yield the expected high scores and a NaN or Inf warning is constantly raised during training.
2020-11-17 11:10:03,030 - INFO - allennlp.training.trainer - Training
0%| | 0/1074 [00:00<?, ?it/s]2020-11-17 11:10:04,066 - WARNING - allennlp.training.util - Metrics with names beginning with "_" will not be logged to the tqdm progress bar.
rel_precision: 0.0000, rel_recall: 0.0000, rel_f1: 0.0000, rel_span_recall: 0.0057, loss: 3.1743 ||: 9%|9 | 99/1074 [01:05<09:49, 1.65it/s]2020-11-17 11:11:10,009 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,009 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,026 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,026 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,029 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,029 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,038 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,038 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,041 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,041 - WARNING - root - NaN or Inf found in input tensor.
rel_precision: 0.0000, rel_recall: 0.0007, rel_f1: 0.0000, rel_span_recall: 0.0060, loss: 2.1448 ||: 19%|#8 | 199/1074 [02:09<08:33, 1.70it/s]2020-11-17 11:12:13,805 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,806 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,818 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,818 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,820 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,820 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,827 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,827 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,829 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,830 - WARNING - root - NaN or Inf found in input tensor.
rel_precision: 0.0000, rel_recall: 0.0005, rel_f1: 0.0000, rel_span_recall: 0.0054, loss: 1.7052 ||: 26%|##5 | 276/1074 [02:58<08:36, 1.54it/s]
...
We run the following command: .\scripts\train\train_kn.sh gpu_id.
This NaN warning appears a couple of times in each epoch. The inter-evaluation also does not improve over the epochs.
Our guess is that the problem might be in the training configs, since the NaN issue might be caused by the older version of allennlp and its tensorboard logging (https://github.com/allenai/allennlp/issues/3116). Did you use the same config as mentioned above for the training of Dygie++ for the KnowledgeNet leaderboard?
Could you help us please? Did we miss anything in the preprocessing that enables successful training?
Hi @schmidek
Thank you a lot for your contribution! We tried to launch your project on KnowledgeNet Dataset (this version: https://github.com/schmidek/dygiepp/tree/multitask/dygie) and encounter with the following issue: the training does not yield the expected high scores and a NaN or Inf warning is constantly raised during training.
The setup is:
We run the following command:
.\scripts\train\train_kn.sh gpu_id.This NaN warning appears a couple of times in each epoch. The inter-evaluation also does not improve over the epochs.
Our guess is that the problem might be in the training configs, since the NaN issue might be caused by the older version of allennlp and its tensorboard logging (https://github.com/allenai/allennlp/issues/3116). Did you use the same config as mentioned above for the training of Dygie++ for the KnowledgeNet leaderboard?
Could you help us please? Did we miss anything in the preprocessing that enables successful training?
Evaluation after the first epoch
> 2020-11-17 11:29:18,090 - INFO - allennlp.training.trainer - Validating > rel_precision: 0.0000, rel_recall: 0.0000, rel_f1: 0.0000, rel_span_recall: 0.0000, loss: 0.2731 ||: 100%|##########| 1022/1022 [01:01<00:00, 16.75it/s] > 2020-11-17 11:30:19,115 - INFO - allennlp.training.tensorboard_writer - Training | Validation > 2020-11-17 11:30:19,116 - INFO - allennlp.training.tensorboard_writer - _arg_id_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,117 - INFO - allennlp.training.tensorboard_writer - _trig_class_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,118 - INFO - allennlp.training.tensorboard_writer - _arg_class_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,118 - INFO - allennlp.training.tensorboard_writer - _arg_class_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,119 - INFO - allennlp.training.tensorboard_writer - gpu_0_memory_MB | 7333.000 | N/A > 2020-11-17 11:30:19,119 - INFO - allennlp.training.tensorboard_writer - _trig_id_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,119 - INFO - allennlp.training.tensorboard_writer - _arg_class_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,120 - INFO - allennlp.training.tensorboard_writer - gpu_4_memory_MB | 4.000 | N/A > 2020-11-17 11:30:19,120 - INFO - allennlp.training.tensorboard_writer - _ner_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,120 - INFO - allennlp.training.tensorboard_writer - cpu_memory_MB | 3344.232 | N/A > 2020-11-17 11:30:19,121 - INFO - allennlp.training.tensorboard_writer - _coref_mention_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,121 - INFO - allennlp.training.tensorboard_writer - _trig_id_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,122 - INFO - allennlp.training.tensorboard_writer - gpu_2_memory_MB | 10289.000 | N/A > 2020-11-17 11:30:19,122 - INFO - allennlp.training.tensorboard_writer - _trig_id_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - gpu_5_memory_MB | 2158.000 | N/A > 2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - gpu_6_memory_MB | 10289.000 | N/A > 2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - gpu_7_memory_MB | 7649.000 | N/A > 2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - _coref_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,124 - INFO - allennlp.training.tensorboard_writer - _trig_class_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,124 - INFO - allennlp.training.tensorboard_writer - _ner_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,125 - INFO - allennlp.training.tensorboard_writer - _coref_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,125 - INFO - allennlp.training.tensorboard_writer - rel_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,126 - INFO - allennlp.training.tensorboard_writer - _args_multiple | 0.000 | 0.000 > 2020-11-17 11:30:19,126 - INFO - allennlp.training.tensorboard_writer - rel_precision | 0.000 | 0.000 > 2020-11-17 11:30:19,127 - INFO - allennlp.training.tensorboard_writer - _trig_class_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,127 - INFO - allennlp.training.tensorboard_writer - _coref_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,128 - INFO - allennlp.training.tensorboard_writer - _ner_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,128 - INFO - allennlp.training.tensorboard_writer - gpu_3_memory_MB | 9653.000 | N/A > 2020-11-17 11:30:19,128 - INFO - allennlp.training.tensorboard_writer - loss | 0.688 | 0.273 > 2020-11-17 11:30:19,129 - INFO - allennlp.training.tensorboard_writer - rel_span_recall | 0.002 | 0.000 > 2020-11-17 11:30:19,129 - INFO - allennlp.training.tensorboard_writer - rel_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,130 - INFO - allennlp.training.tensorboard_writer - _arg_id_recall | 0.000 | 0.000 > 2020-11-17 11:30:19,130 - INFO - allennlp.training.tensorboard_writer - _arg_id_f1 | 0.000 | 0.000 > 2020-11-17 11:30:19,130 - INFO - allennlp.training.tensorboard_writer - gpu_1_memory_MB | 9653.000 | N/A > 2020-11-17 11:30:21,546 - INFO - allennlp.training.checkpointer - Best validation performance so far. Copying weights to './models/kn/best.th'. > 2020-11-17 11:30:22,486 - INFO - allennlp.training.trainer - Epoch duration: 0:12:33.792339 > 2020-11-17 11:30:22,488 - INFO - allennlp.training.trainer - Estimated training time remaining: 10:15:35 > 2020-11-17 11:30:22,488 - INFO - allennlp.training.trainer - Epoch 1/49Evaluation after epoch 22 (early-stopping)
> rel_precision: 0.0001, rel_recall: 0.0269, rel_f1: 0.0001, rel_span_recall: 0.0370, loss: 0.4834 ||: 100%|##########| > 1022/1022 [00:57<00:00, 17.70it/s] > 2020-11-17 15:06:06,074 - INFO - allennlp.training.trainer - Ran out of patience. Stopping training. > 2020-11-17 15:06:06,076 - INFO - allennlp.training.checkpointer - loading best weights > 2020-11-17 15:06:06,349 - INFO - allennlp.commands.train - To evaluate on the test set after training, pass the 'evaluate_on_test' flag, or use the 'allennlp evaluate' command. > 2020-11-17 15:06:06,350 - INFO - allennlp.models.archival - archiving weights and vocabulary to ./models/kn/model.tar.gz > 2020-11-17 15:06:30,631 - INFO - allennlp.common.util - Metrics: { > "best_epoch": 8, > "peak_cpu_memory_MB": 5457.956, > "peak_gpu_0_memory_MB": 7333, > "peak_gpu_1_memory_MB": 9653, > "peak_gpu_2_memory_MB": 10289, > "peak_gpu_3_memory_MB": 9653, > "peak_gpu_4_memory_MB": 10439, > "peak_gpu_5_memory_MB": 1079, > "peak_gpu_6_memory_MB": 10289, > "peak_gpu_7_memory_MB": 7649, > "training_duration": "3:17:07.685956", > "training_start_epoch": 0, > "training_epochs": 22, > "epoch": 22, > "training__coref_precision": 0.0, > "training__coref_recall": 0.0, > "training__coref_f1": 0.0, > "training__coref_mention_recall": 0.0, > "training__ner_precision": 0.0, > "training__ner_recall": 0.0, > "training__ner_f1": 0.0, > "training_rel_precision": 3.5733284070843595e-05, > "training_rel_recall": 0.013081574725410341, > "training_rel_f1": 7.127188374614434e-05, > "training_rel_span_recall": 0.016413673947920523, > "training__trig_id_precision": 0, > "training__trig_id_recall": 0, > "training__trig_id_f1": 0, > "training__trig_class_precision": 0, > "training__trig_class_recall": 0, > "training__trig_class_f1": 0, > "training__arg_id_precision": 0, > "training__arg_id_recall": 0, > "training__arg_id_f1": 0, > "training__arg_class_precision": 0, > "training__arg_class_recall": 0, > "training__arg_class_f1": 0, > "training__args_multiple": 0, > "training_loss": 0.010883575075255758, > "training_cpu_memory_MB": 5334.428, > "training_gpu_0_memory_MB": 7333, > "training_gpu_1_memory_MB": 9653, > "training_gpu_2_memory_MB": 10289, > "training_gpu_3_memory_MB": 9653, > "training_gpu_4_memory_MB": 10439, > "training_gpu_5_memory_MB": 1079, > "training_gpu_6_memory_MB": 10289, > "training_gpu_7_memory_MB": 7649, > "validation__coref_precision": 0.0, > "validation__coref_recall": 0.0, > "validation__coref_f1": 0.0, > "validation__coref_mention_recall": 0.0, > "validation__ner_precision": 0.0, > "validation__ner_recall": 0.0, > "validation__ner_f1": 0.0, > "validation_rel_precision": 5.676047081316843e-05, > "validation_rel_recall": 0.02132435465768799, > "validation_rel_f1": 0.0001132195774526191, > "validation_rel_span_recall": 0.0306771417882529, > "validation__trig_id_precision": 0, > "validation__trig_id_recall": 0, > "validation__trig_id_f1": 0, > "validation__trig_class_precision": 0, > "validation__trig_class_recall": 0, > "validation__trig_class_f1": 0, > "validation__arg_id_precision": 0, > "validation__arg_id_recall": 0, > "validation__arg_id_f1": 0, > "validation__arg_class_precision": 0, > "validation__arg_class_recall": 0, > "validation__arg_class_f1": 0, > "validation__args_multiple": 0, > "validation_loss": 0.3997326233545447, > "best_validation__coref_precision": 0.0, > "best_validation__coref_recall": 0.0, > "best_validation__coref_f1": 0.0, > "best_validation__coref_mention_recall": 0.0, > "best_validation__ner_precision": 0.0, > "best_validation__ner_recall": 0.0, > "best_validation__ner_f1": 0.0, > "best_validation_rel_precision": 0.00013013713035796913, > "best_validation_rel_recall": 0.037037037037037035, > "best_validation_rel_f1": 0.00025936293651240686, > "best_validation_rel_span_recall": 0.04526748971193416, > "best_validation__trig_id_precision": 0, > "best_validation__trig_id_recall": 0, > "best_validation__trig_id_f1": 0, > "best_validation__trig_class_precision": 0, > "best_validation__trig_class_recall": 0, > "best_validation__trig_class_f1": 0, > "best_validation__arg_id_precision": 0, > "best_validation__arg_id_recall": 0, > "best_validation__arg_id_f1": 0, > "best_validation__arg_class_precision": 0, > "best_validation__arg_class_recall": 0, > "best_validation__arg_class_f1": 0, > "best_validation__args_multiple": 0, > "best_validation_loss": 0.15924045607033888 > }