search

yao8839836 / kg-bert

KG-BERT: BERT for Knowledge Graph Completion
Apache License 2.0
679 stars 141 forks source link

WARNING - pytorch_pretrained_bert.optimization - Training beyond specified 't_total'. Learning rate multiplier set to 0.0. Please set 't_total' of WarmupLinearSchedule correctly. #5

Closed moh-yani closed 4 years ago

moh-yani commented 4 years ago

when I execute

python run_bert_triple_classifier.py --task_name kg
--do_train
--do_eval --do_predict --data_dir ./data/FB15K --bert_model bert-base-cased --max_seq_length 200 --train_batch_size 12 --learning_rate 5e-5 --num_train_epochs 3.0 --output_dir ./output_FB15K/
--gradient_accumulation_steps 1 --eval_batch_size 12

Here the information of training: 13:52:37 - INFO - main - Running training 13:52:37 - INFO - main - Num examples = 966284 13:52:37 - INFO - main - Batch size = 12 13:52:37 - INFO - main - Num steps = 241569 Epoch: 0%| | 0/3 [00:00<?, ?it/s] Iteration: 17%|██████▎ | 13815/80524 [1:37:39<7:50:50, 2. etc.

At the final epoch of training I get the warning and error such as below:

18:08:12 - WARNING - pytorch_pretrained_bert.optimization - Training beyond specified 't_total'. Learning rate multiplier set to 0.0. Please set 't_total' of WarmupLinearSchedule correctly.

18:08:12 - WARNING - pytorch_pretrained_bert.optimization - Training beyond specified 't_total'. Learning rate multiplier set to 0.0. Please set 't_total' of WarmupLinearSchedule correctly.

18:08:13 - INFO - pytorch_pretrained_bert.modeling - loading archive file ./output_FB15K/ 18:08:13 - INFO - pytorch_pretrained_bert.modeling - Model config { "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072,
"max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12,
"type_vocab_size": 2, "vocab_size": 28996
}

18:08:15 - INFO - pytorch_pretrained_bert.tokenization - loading vocabulary file ./output_FB15K/vocab.txt Traceback (most recent call last): File "run_bert_triple_classifier.py", line 858, in main()
File "run_bert_triple_classifier.py", line 708, in main eval_examples = processor.get_dev_examples(args.data_dir)
File "run_bert_triple_classifier.py", line 135, in get_dev_examples self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev", data_dir) File "run_bert_triple_classifier.py", line 207, in _create_examples triple_label = line[3]
IndexError: list index out of range

Anyone know this problem and the solution?

yao8839836 commented 4 years ago

@moh-yani

The error is due to the fact that test triples of FB15K don't have triple labels (1 or -1) because they are all correct triples, when performing triple classification, your dev.tsv and test.tsv should be like /data/FB13/test.tsv which has 4 columns, you also need negative (incorrect) triples.

moh-yani commented 4 years ago

Nice, thank you @yao8839836.

One question: in https://github.com/yao8839836/kg-bert, you use FB15K data for Relation Prediction and WN12RR data for Link Prediction. The question is, could I use FB13 data for both Relation Prediction and Link Prediction without any changes in the data (train.tsv, dev.tsv, and test.tsv)? If I need to changes for any data, which data should be changed?

Big thanks.

yao8839836 commented 4 years ago

@moh-yani

You can perform link prediction and relation prediction on FB13 by removing negative triples (labeled -1) in dev.tsv, and test.tsv.

moh-yani commented 4 years ago

How about 4th column of dev.tsv and test.tsv in FB13? should I remove it too?

yao8839836 commented 4 years ago

@moh-yani

You can keep it, because run_bert_link_prediction.py and run_bert_relation_prediction.py only use the first 3 columns.

By the way , FB13 was made to evaluate triple classifiers in https://nlp.stanford.edu/pubs/SocherChenManningNg_NIPS2013.pdf, I didn't see any previous paper which performs link prediction and relation prediction on FB13.

moh-yani commented 4 years ago

Great, thank your very much.