Slow validation and test speed on large knowledge graphs

[zhongpeixiang]

Hi,

I experimented with TransE and TransH on ConceptNet (2.2M entities, 37 relations and 4.8M triplets).

The training speed looks good. But the validation speed (using 16 cores CPU or 1080Ti and V100 GPUs) is slow, e.g., 6 mins per 1000 triplets for TransH using V100 GPU. Testing on 50K triplets costs nearly 5 hours, which makes early_stopping not feasible.

The training and testing configs are given below:

import config
from  models import *
import json
import os 
os.environ['CUDA_VISIBLE_DEVICES']='0'
con = config.Config()
con.set_use_gpu(True)
con.set_work_threads(6)
con.set_in_path("../data/KB/ConceptNet-KE/")
con.set_train_times(1000)
con.set_nbatches(100)   
con.set_alpha(0.001)
con.set_bern(0)
con.set_dimension(100)
con.set_margin(1.0)
con.set_ent_neg_rate(1)
con.set_rel_neg_rate(0)
con.set_opt_method("SGD")
con.set_save_steps(100)
con.set_valid_steps(100)
con.set_early_stopping_patience(10)
con.set_checkpoint_dir("./checkpoint")
con.set_result_dir("./result")
con.set_test_link(True)
con.set_test_triple(True)
con.init()
con.set_train_model(TransH)
con.train()

import config
from models import *
import json
import os 
os.environ['CUDA_VISIBLE_DEVICES']='0'
con = config.Config()
con.set_use_gpu(True)
con.set_in_path("../data/KB/ConceptNet-KE/")
con.set_result_dir("./result")
con.set_test_link(True)
con.set_test_triple(True)
con.init()
con.set_test_model(TransH)
con.test()

[chenweize1998]

It's a problem of our implementation for validation and testing process. We calculate the scores with GPU, but do the validation and testing part with C code using CPU. It's much faster on common datasets like FB15K237. But it seems that the approach is not suitable for large datasets ConceptNet. The easiest way that you can try is to open more threads by set_work_threads method. It's not the best way, but at least it will speed up the validation and testing process.

[zhongpeixiang]

It seems that when I run test_transH.py with use_gpu=True, the multi-threading does work.

[zxy951005]

Dear ZhongPeixiang: Does the result(hit@1, hit@3, and hit@10) of the TransE model on the ConceptNet dataset perform well?

[zhongpeixiang]

Dear ZhongPeixiang: Does the result(hit@1, hit@3, and hit@10) of the TransE model on the ConceptNet dataset perform well?

Can’t remember the exact results. The hit@1 is bad but MRR is around 0.2 to 0.3 on a filtered ConceptNet. The binary triplet classification accuracy is around 0.9.