The pre-train result is unsatisfactory with default parameter.

[G-B-Y]

In pre-training of umls with default parameter, the result is as followed:

{'Test/hits1': 0.00202020202020202, 'Test/hits10': 0.06868686868686869, 'Test/hits20': 0.13198653198653199, 'Test/hits3': 0.017508417508417508, 'Test/mean_rank': 70.0895622895623, 'Test/mrr': 0.034388057056645285}

In pre-training of WN18RR with default parameter, the result is as followed:

{'Test/hits1': 3.552610835907126e-05, 'Test/hits10': 0.00027088657623791835, 'Test/hits20': 0.0005639769702002563, 'Test/hits3': 7.993374380791033e-05, 'Test/mean_rank': 20417.76962206882, 'Test/mrr': 0.00030194088791004475}

[bizhen46766]

Hi, you should continue to run the next train step. The outputs in the pre-trainings stage are intermediate results.

[G-B-Y]

Sure, but the result is also unsatisfactory with default parameter.

In fine-tuning of WN18RR with default parameter, the result is as followed:

{'Test/hits1': 0.0001595405232929164, 'Test/hits10': 0.0001595405232929164, 'Test/hits20': 0.0006381620931716656, 'Test/hits3': 0.0001595405232929164, 'Test/mean_rank': 20385.440810465858, 'Test/mrr': 0.0003705603592570774}

In fine-tuning of umls with default parameter, the result is as followed:

{'Test/hits1': 0.01739788199697428, 'Test/hits10': 0.09379727685325265, 'Test/hits20': 0.17170953101361575, 'Test/hits3': 0.03555219364599092, 'Test/mean_rank': 58.33888048411498, 'Test/mrr': 0.05413661936632273}

[bizhen46766]

Hi, the result seems quite strange, do you correctly load the pre-trained weights?

[G-B-Y]

Hi, I only use the default parameter.

For the pre-training of umls:

nohup python -u main.py --gpus "0," --max_epochs=20  --num_workers=32 \
   --model_name_or_path  bert-base-uncased \
   --accumulate_grad_batches 1 \
   --model_class BertKGC \
   --batch_size 128 \
   --pretrain 1 \
   --bce 0 \
   --check_val_every_n_epoch 1 \
   --overwrite_cache \
   --data_dir /mine/Relphormer/dataset/umls \
   --eval_batch_size 256 \
   --max_seq_length 64 \
   --lr 1e-4 \
   >logs/pretrain_umls.log 2>&1 &

For the pre-training of WN18RR:

nohup python -u main.py --gpus "0," --max_epochs=15  --num_workers=32 \
   --model_name_or_path  bert-base-uncased \
   --accumulate_grad_batches 1 \
   --bce 0 \
   --model_class BertKGC \
   --batch_size 128 \
   --pretrain 1 \
   --check_val_every_n_epoch 1 \
   --data_dir /mine/Relphormer/dataset/WN18RR \
   --overwrite_cache \
   --eval_batch_size 256 \
   --precision 16 \
   --max_seq_length 32 \
   --lr 1e-4 \
   >logs/pretrain_wn18rr.log 2>&1 &

[bizhen46766]

There seems to be no problem for the parameters, perhaps you can check the model path again and re-train the whole model.

[G-B-Y]

Thanks for answer. I will try it again.

[bizhen46766]

Hi! we have updated the project and fixed the above running problem. Sorry for the inconvenience.

[zxlzr]

We have also released the pre-trained checkpoints, have a try.