Closed G-B-Y closed 1 year ago
Hi, you should continue to run the next train step. The outputs in the pre-trainings stage are intermediate results.
Sure, but the result is also unsatisfactory with default parameter.
In fine-tuning of WN18RR with default parameter, the result is as followed:
{'Test/hits1': 0.0001595405232929164, 'Test/hits10': 0.0001595405232929164, 'Test/hits20': 0.0006381620931716656, 'Test/hits3': 0.0001595405232929164, 'Test/mean_rank': 20385.440810465858, 'Test/mrr': 0.0003705603592570774}
In fine-tuning of umls with default parameter, the result is as followed:
{'Test/hits1': 0.01739788199697428, 'Test/hits10': 0.09379727685325265, 'Test/hits20': 0.17170953101361575, 'Test/hits3': 0.03555219364599092, 'Test/mean_rank': 58.33888048411498, 'Test/mrr': 0.05413661936632273}
Hi, the result seems quite strange, do you correctly load the pre-trained weights?
Hi, I only use the default parameter.
For the pre-training of umls:
nohup python -u main.py --gpus "0," --max_epochs=20 --num_workers=32 \
--model_name_or_path bert-base-uncased \
--accumulate_grad_batches 1 \
--model_class BertKGC \
--batch_size 128 \
--pretrain 1 \
--bce 0 \
--check_val_every_n_epoch 1 \
--overwrite_cache \
--data_dir /mine/Relphormer/dataset/umls \
--eval_batch_size 256 \
--max_seq_length 64 \
--lr 1e-4 \
>logs/pretrain_umls.log 2>&1 &
For the pre-training of WN18RR:
nohup python -u main.py --gpus "0," --max_epochs=15 --num_workers=32 \
--model_name_or_path bert-base-uncased \
--accumulate_grad_batches 1 \
--bce 0 \
--model_class BertKGC \
--batch_size 128 \
--pretrain 1 \
--check_val_every_n_epoch 1 \
--data_dir /mine/Relphormer/dataset/WN18RR \
--overwrite_cache \
--eval_batch_size 256 \
--precision 16 \
--max_seq_length 32 \
--lr 1e-4 \
>logs/pretrain_wn18rr.log 2>&1 &
There seems to be no problem for the parameters, perhaps you can check the model path again and re-train the whole model.
Thanks for answer. I will try it again.
Hi! we have updated the project and fixed the above running problem. Sorry for the inconvenience.
We have also released the pre-trained checkpoints, have a try.
In pre-training of umls with default parameter, the result is as followed:
{'Test/hits1': 0.00202020202020202, 'Test/hits10': 0.06868686868686869, 'Test/hits20': 0.13198653198653199, 'Test/hits3': 0.017508417508417508, 'Test/mean_rank': 70.0895622895623, 'Test/mrr': 0.034388057056645285}
In pre-training of WN18RR with default parameter, the result is as followed:
{'Test/hits1': 3.552610835907126e-05, 'Test/hits10': 0.00027088657623791835, 'Test/hits20': 0.0005639769702002563, 'Test/hits3': 7.993374380791033e-05, 'Test/mean_rank': 20417.76962206882, 'Test/mrr': 0.00030194088791004475}