Question about table 6

[ty4b112]

I want to know the fairness of this experiment. The article describes how to handle test sets and training sets. But Neural-LP reasons based on known facts and rules. The facts is splited from the initial train data and contains tuples share entities with selected test tuples. In order to ensure fairness, different models should use the same data. I want to know the data you use to train TransE because I think it is not the union of facts and train for Neural-LP. What's your interpretation of the fairness of this experiment?

[fanyangxyz]

The training data for TransE is the union of facts and train. I think the experiment is fair.

[ty4b112]

Thank you for your reply. I am still confused that if the training data for TransE is the union of facts and train, it means it contains tuples that share entities with selected test tuples, why Hits@10 drops to near zero. I tried to reproduce this experiment on WN18 but the results are different. Have you done some processing on the facts?

[fanyangxyz]

For the inductive setting, there is a filtering step in preprocessing. This does not conflict with "training data for xxxx is the union of facts and train", since train and facts are further split of the filtered train.

[ty4b112]

Dose the "filtering step" equals "filter the train set by excluding any tuples that share entities with selected test tuples"? If facts used during testing is from the filtered train, I think the TensorLog opreation will not work.

[fanyangxyz]

Yes, this is how the filtering step works. Not sure about your statement on TensorLog, could you elaborate?

[ty4b112]

To my understanding, TensorLog use known facts constrcut sparse matrices an do Matrix multiplication to reason. If the facts don't share entities with testing data, the result of matrix multipliction will be zero vector.

[fanyangxyz]

I think you're right. It's possible that in the inductive setting, the training data for TransE is not the union of train and facts (but for other setting this is the case). Given this, I still think the experiment setting is fair.

[ty4b112]

I want to do some work based on Neural LP. Neural LP is able to evaluate tuples contains entities that are not in the training data, which is a good property. I want my model has the same. But I can't design an experiment to compared with other models. Why do you think your experiment setting is fair?

[fanyangxyz]

I guess the key point is that if compare with a model that is not designed for inductive setting, some modifications in experiment settings are needed. I feel as long as the modifications are justified, for example to make the setting inductive, it is fair.

[ty4b112]

Thanks, your explanation is very helpful.