thunlp / OpenKE

An Open-Source Package for Knowledge Embedding (KE)
3.83k stars 985 forks source link

Slow validation and test speed on large knowledge graphs #197

Closed zhongpeixiang closed 4 years ago

zhongpeixiang commented 5 years ago

Hi,

I experimented with TransE and TransH on ConceptNet (2.2M entities, 37 relations and 4.8M triplets).

The training speed looks good. But the validation speed (using 16 cores CPU or 1080Ti and V100 GPUs) is slow, e.g., 6 mins per 1000 triplets for TransH using V100 GPU. Testing on 50K triplets costs nearly 5 hours, which makes early_stopping not feasible.

The training and testing configs are given below:

import config
from  models import *
import json
import os 
os.environ['CUDA_VISIBLE_DEVICES']='0'
con = config.Config()
con.set_use_gpu(True)
con.set_work_threads(6)
con.set_in_path("../data/KB/ConceptNet-KE/")
con.set_train_times(1000)
con.set_nbatches(100)   
con.set_alpha(0.001)
con.set_bern(0)
con.set_dimension(100)
con.set_margin(1.0)
con.set_ent_neg_rate(1)
con.set_rel_neg_rate(0)
con.set_opt_method("SGD")
con.set_save_steps(100)
con.set_valid_steps(100)
con.set_early_stopping_patience(10)
con.set_checkpoint_dir("./checkpoint")
con.set_result_dir("./result")
con.set_test_link(True)
con.set_test_triple(True)
con.init()
con.set_train_model(TransH)
con.train()
import config
from models import *
import json
import os 
os.environ['CUDA_VISIBLE_DEVICES']='0'
con = config.Config()
con.set_use_gpu(True)
con.set_in_path("../data/KB/ConceptNet-KE/")
con.set_result_dir("./result")
con.set_test_link(True)
con.set_test_triple(True)
con.init()
con.set_test_model(TransH)
con.test()
chenweize1998 commented 5 years ago

It's a problem of our implementation for validation and testing process. We calculate the scores with GPU, but do the validation and testing part with C code using CPU. It's much faster on common datasets like FB15K237. But it seems that the approach is not suitable for large datasets ConceptNet. The easiest way that you can try is to open more threads by set_work_threads method. It's not the best way, but at least it will speed up the validation and testing process.

zhongpeixiang commented 5 years ago

It seems that when I run test_transH.py with use_gpu=True, the multi-threading does work.

zxy951005 commented 3 years ago

Dear ZhongPeixiang: Does the result(hit@1, hit@3, and hit@10) of the TransE model on the ConceptNet dataset perform well?

zhongpeixiang commented 3 years ago

Dear ZhongPeixiang: Does the result(hit@1, hit@3, and hit@10) of the TransE model on the ConceptNet dataset perform well?

Can’t remember the exact results. The hit@1 is bad but MRR is around 0.2 to 0.3 on a filtered ConceptNet. The binary triplet classification accuracy is around 0.9.