Is the data in the paper inconsistent with the data provided by the warehouse?

[zhiweihu1103]

I have performed statistics on all the version data provided in the warehouse, and found that there is some inconsistency with the statistical results in Table 2 of the paper? Can you give a brief explanation? Paper Table 2:

My statistics table:

The red and bold parts are inconsistent data. Looking forward to your reply!!!

The statistics code I used is :

    root_path = 'data/WN18RR_v2_ind'
    file_list = [root_path + '/train.txt', root_path + '/valid.txt', root_path + '/test.txt']
    relation_list = []
    entity_list = []
    count = 0
    for file_path in file_list:
        with open(file_path) as f:
            file_data = [line.split() for line in f.read().split('\n')[:-1]]
            count = count + len(file_data)
            for triplet in file_data:
                if triplet[0] not in entity_list:
                    entity_list.append(triplet[0])
                if triplet[2] not in entity_list:
                    entity_list.append(triplet[2])
                if triplet[1] not in relation_list:
                    relation_list.append(triplet[1])
    print(root_path[root_path.rfind('/')+1:])
    print(len(relation_list))
    print(len(entity_list))
    print(count)

[JiajunChen98]

Thank you for pointing out this. This issue has been addressed by the author of GraIL (https://github.com/kkteru/grail/issues/6). We will update the paper accordingly.

[zhiweihu1103]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

[JiajunChen98]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

The train.txt in the _ind datasets can be seen as the input of the model when testing. The model predict the query in the test set (in the _ind datasets) based on the graph constructed by train.txt in the *_ind datasets.

[zhiweihu1103]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

The train.txt in the _ind datasets can be seen as the input of the model when testing. The model predict the query in the test set (in the _ind datasets) based on the graph constructed by train.txt in the *_ind datasets.

In the _ind datasets, there are the same entities between train.txt and test.txt. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the _ind datasets)? Train.txt is only used to negatively sample the test.txt triples and obtain the corresponding closed subgraph of the triplet on the test (in the *_ind datasets). Is this understanding correct?

[JiajunChen98]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

The train.txt in the _ind datasets can be seen as the input of the model when testing. The model predict the query in the test set (in the _ind datasets) based on the graph constructed by train.txt in the *_ind datasets.

In the _ind datasets, there are the same entities between train.txt and test.txt. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the _ind datasets)? Train.txt is only used to negatively sample the test.txt triples and obtain the corresponding closed subgraph of the triplet on the test (in the *_ind datasets). Is this understanding correct?

Yes. This is right. All the files in (such as fb237_v1) are used when training, and the files in _ind (such as fb237_v1_ind) are used when testing. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the *_ind datasets).

[zhiweihu1103]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

The train.txt in the _ind datasets can be seen as the input of the model when testing. The model predict the query in the test set (in the _ind datasets) based on the graph constructed by train.txt in the *_ind datasets.

In the _ind datasets, there are the same entities between train.txt and test.txt. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the _ind datasets)? Train.txt is only used to negatively sample the test.txt triples and obtain the corresponding closed subgraph of the triplet on the test (in the *_ind datasets). Is this understanding correct?

Yes. This is right. All the files in (such as fb237_v1) are used when training, and the files in _ind (such as fb237_v1_ind) are used when testing. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the *_ind datasets).

What is the purpose of the existence of valid.txt in the *_ind datasets? I found no trace of valid.txt in test_auc.py. Thx.

[JiajunChen98]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

The train.txt in the _ind datasets can be seen as the input of the model when testing. The model predict the query in the test set (in the _ind datasets) based on the graph constructed by train.txt in the *_ind datasets.

In the _ind datasets, there are the same entities between train.txt and test.txt. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the _ind datasets)? Train.txt is only used to negatively sample the test.txt triples and obtain the corresponding closed subgraph of the triplet on the test (in the *_ind datasets). Is this understanding correct?

Yes. This is right. All the files in (such as fb237_v1) are used when training, and the files in _ind (such as fb237_v1_ind) are used when testing. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the *_ind datasets).

What is the purpose of the existence of valid.txt in the *_ind datasets? I found no trace of valid.txt in test_auc.py. Thx.

It's not used when testing. I guess the purpose is to keep the same dataset split strategy with the files in *. This dataset is proposed by the authors of GraIL, you can asks the proposers for more details.

[zhiweihu1103]

Thank you for your reply. Another question is, in your README file "We run each experiment five times and report the mean results", that means you test the performance in the _ind datasets in the test.txt file? This is the understand from the test_auc.py. I want to ask what is the meaning of existence of train.txt and valid.txt in the _ind datasets, they all don't participate in the training or testing process. Looking forward to your reply.

The train.txt in the _ind datasets can be seen as the input of the model when testing. The model predict the query in the test set (in the _ind datasets) based on the graph constructed by train.txt in the *_ind datasets.

In the _ind datasets, there are the same entities between train.txt and test.txt. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the _ind datasets)? Train.txt is only used to negatively sample the test.txt triples and obtain the corresponding closed subgraph of the triplet on the test (in the *_ind datasets). Is this understanding correct?

Yes. This is right. All the files in (such as fb237_v1) are used when training, and the files in _ind (such as fb237_v1_ind) are used when testing. The results of AUC_PR and Hit@1 in the paper are obtained from test.txt (in the *_ind datasets).

What is the purpose of the existence of valid.txt in the *_ind datasets? I found no trace of valid.txt in test_auc.py. Thx.

It's not used when testing. I guess the purpose is to keep the same dataset split strategy with the files in *. This dataset is proposed by the authors of GraIL, you can asks the proposers for more details.

Thank you very much for your help. Best wishes to you.