Hello, may I ask why dev can't be used to verify the validation set using the model you gave me? Keyerror will be reported. If I ignore keyerror, AttributeError will also be generated #8
08/30/2022 21:51:23 - INFO - __main__ - model_type - longformer
08/30/2022 21:51:23 - INFO - __main__ - model_name_or_path - /home/tanner/git/s2e-coref/model
08/30/2022 21:51:23 - INFO - __main__ - output_dir - output
08/30/2022 21:51:23 - INFO - __main__ - train_file_cache - /home/tanner/conll/conll-2012/english/train.english.4096.pkl
08/30/2022 21:51:23 - INFO - __main__ - predict_file_cache - /home/tanner/conll/conll-2012/english/test.english.4096.pkl
08/30/2022 21:51:23 - INFO - __main__ - train_file - /home/tanner/conll/conll-2012/english/train.english.jsonlines
08/30/2022 21:51:23 - INFO - __main__ - predict_file - /home/tanner/conll/conll-2012/english/test.english.jsonlines
08/30/2022 21:51:23 - INFO - __main__ - config_name - allenai/longformer-large-4096
08/30/2022 21:51:23 - INFO - __main__ - tokenizer_name - allenai/longformer-large-4096
08/30/2022 21:51:23 - INFO - __main__ - cache_dir - cache
08/30/2022 21:51:23 - INFO - __main__ - max_seq_length - 4096
08/30/2022 21:51:23 - INFO - __main__ - do_train - False
08/30/2022 21:51:23 - INFO - __main__ - do_eval - True
08/30/2022 21:51:23 - INFO - __main__ - do_lower_case - False
08/30/2022 21:51:23 - INFO - __main__ - nonfreeze_params - None
08/30/2022 21:51:23 - INFO - __main__ - learning_rate - 1e-05
08/30/2022 21:51:23 - INFO - __main__ - head_learning_rate - 0.0003
08/30/2022 21:51:23 - INFO - __main__ - dropout_prob - 0.3
08/30/2022 21:51:23 - INFO - __main__ - gradient_accumulation_steps - 1
08/30/2022 21:51:23 - INFO - __main__ - weight_decay - 0.01
08/30/2022 21:51:23 - INFO - __main__ - adam_beta1 - 0.9
08/30/2022 21:51:23 - INFO - __main__ - adam_beta2 - 0.98
08/30/2022 21:51:23 - INFO - __main__ - adam_epsilon - 1e-06
08/30/2022 21:51:23 - INFO - __main__ - num_train_epochs - 129.0
08/30/2022 21:51:23 - INFO - __main__ - warmup_steps - 5600
08/30/2022 21:51:23 - INFO - __main__ - logging_steps - 500
08/30/2022 21:51:23 - INFO - __main__ - eval_steps - 1000
08/30/2022 21:51:23 - INFO - __main__ - save_steps - 3000
08/30/2022 21:51:23 - INFO - __main__ - no_cuda - False
08/30/2022 21:51:23 - INFO - __main__ - overwrite_output_dir - False
08/30/2022 21:51:23 - INFO - __main__ - seed - 42
08/30/2022 21:51:23 - INFO - __main__ - local_rank - -1
08/30/2022 21:51:23 - INFO - __main__ - amp - True
08/30/2022 21:51:23 - INFO - __main__ - fp16_opt_level - O1
08/30/2022 21:51:23 - INFO - __main__ - max_span_length - 30
08/30/2022 21:51:23 - INFO - __main__ - top_lambda - 0.4
08/30/2022 21:51:23 - INFO - __main__ - max_total_seq_len - 5000
08/30/2022 21:51:23 - INFO - __main__ - experiment_name - eval_model
08/30/2022 21:51:23 - INFO - __main__ - normalise_loss - True
08/30/2022 21:51:23 - INFO - __main__ - ffnn_size - 3072
08/30/2022 21:51:23 - INFO - __main__ - save_if_best - True
08/30/2022 21:51:23 - INFO - __main__ - batch_size_1 - False
08/30/2022 21:51:23 - INFO - __main__ - tensorboard_dir - output/tb
08/30/2022 21:51:23 - INFO - __main__ - conll_path_for_eval - /home/tanner/conll/conll-2012/english/dev.english.v4_gold_conll
08/30/2022 21:51:23 - INFO - __main__ - n_gpu - 0
08/30/2022 21:51:23 - INFO - __main__ - device - cpu
Writing output/meta.json
08/30/2022 21:51:23 - INFO - __main__ - Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, amp training: True
08/30/2022 21:51:34 - INFO - __main__ - Training/evaluation parameters Namespace(adam_beta1=0.9, adam_beta2=0.98, adam_epsilon=1e-06, amp=True, batch_size_1=False, cache_dir='cache', config_name='allenai/longformer-large-4096', conll_path_for_eval='/home/tanner/conll/conll-2012/english/dev.english.v4_gold_conll', device=device(type='cpu'), do_eval=True, do_lower_case=False, do_train=False, dropout_prob=0.3, eval_steps=1000, experiment_name='eval_model', ffnn_size=3072, fp16_opt_level='O1', gradient_accumulation_steps=1, head_learning_rate=0.0003, learning_rate=1e-05, local_rank=-1, logging_steps=500, max_seq_length=4096, max_span_length=30, max_total_seq_len=5000, model_name_or_path='/home/tanner/git/s2e-coref/model', model_type='longformer', n_gpu=0, no_cuda=False, nonfreeze_params=None, normalise_loss=True, num_train_epochs=129.0, output_dir='output', overwrite_output_dir=False, predict_file='/home/tanner/conll/conll-2012/english/test.english.jsonlines', predict_file_cache='/home/tanner/conll/conll-2012/english/test.english.4096.pkl', save_if_best=True, save_steps=3000, seed=42, tensorboard_dir='output/tb', tokenizer_name='allenai/longformer-large-4096', top_lambda=0.4, train_file='/home/tanner/conll/conll-2012/english/train.english.jsonlines', train_file_cache='/home/tanner/conll/conll-2012/english/train.english.4096.pkl', warmup_steps=5600, weight_decay=0.01)
08/30/2022 21:51:34 - INFO - data - Reading dataset from /home/tanner/conll/conll-2012/english/test.english.jsonlines
08/30/2022 21:51:47 - INFO - data - Finished preprocessing Coref dataset. 348 examples were extracted, 0 were filtered due to sequence length.
/home/tanner/git/s2e-coref/env-3.8/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:1767: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
warnings.warn(
08/30/2022 21:51:47 - INFO - eval - ***** Running evaluation final_evaluation *****
08/30/2022 21:51:47 - INFO - eval - Examples number: 348
08/30/2022 22:00:59 - INFO - eval - ***** Eval results final_evaluation *****
08/30/2022 22:00:59 - INFO - eval - loss = 0.421
08/30/2022 22:00:59 - INFO - eval - post pruning mention precision = 0.248
08/30/2022 22:00:59 - INFO - eval - post pruning mention recall = 0.961
08/30/2022 22:00:59 - INFO - eval - post pruning mention f1 = 0.395
08/30/2022 22:00:59 - INFO - eval - mention precision = 0.893
08/30/2022 22:00:59 - INFO - eval - mention recall = 0.878
08/30/2022 22:00:59 - INFO - eval - mention f1 = 0.886
08/30/2022 22:00:59 - INFO - eval - precision = 0.812
08/30/2022 22:00:59 - INFO - eval - recall = 0.795
08/30/2022 22:00:59 - INFO - eval - f1 = 0.804
and an error:
Traceback (most recent call last):
File "run_coref.py", line 155, in <module>
main()
File "run_coref.py", line 147, in main
result = evaluator.evaluate(model, prefix="final_evaluation", official=True)
File "/home/tanner/git/s2e-coref/eval.py", line 138, in evaluate
conll_results = evaluate_conll(self.args.conll_path_for_eval, doc_to_prediction, doc_to_subtoken_map)
File "/home/tanner/git/s2e-coref/conll.py", line 98, in evaluate_conll
output_conll(gold_file, prediction_file, predictions, subtoken_maps)
File "/home/tanner/git/s2e-coref/conll.py", line 47, in output_conll
start_map, end_map, word_map = prediction_map[doc_key]
KeyError: 'bc/cctv/00/cctv_0000_0'
I am looking into this further and post back my solution if I find one. In the meanwhile, I wanted to check if @zyt888 (or @yuvalkirstain or @oriram) know what the problem seems to be.
Just to elaborate on what I perceive to be @zyt888's issue. When I run the following with
export SPLIT_FOR_EVAL=test:I get the following output:
and an error:
I am looking into this further and post back my solution if I find one. In the meanwhile, I wanted to check if @zyt888 (or @yuvalkirstain or @oriram) know what the problem seems to be.