search

soledad921 / ATISE

The source code of ISWC2020 paper "Temporal Knowledge Graph completion based on time series Gaussian embedding " and COLING2020 paper "TeRo: A Time-aware Knowledge Graph Embedding via Temporal Rotation "
MIT License
63 stars 10 forks source link

About year_list #2

Open xjzhao001 opened 4 years ago

xjzhao001 commented 4 years ago

Dear soledad921： Thank you very much for sharing. I have a question, when you are processing the YAGO dataset, I noticed that in Dataset_YG.py, you commented out the two lines of code: 104 # for i in range(start,end): 105 # year_list.append(i) I also think that the years between the start time and the end time should be counted in the year_list, but this will greatly increase the year_list, and it will also lead to a sharp increase in the effective facts in each year. What I want to know is, what made you add those two lines of code and then comment them out?

soledad921 commented 4 years ago

Dear soledad921： Thank you very much for sharing. I have a question, when you are processing the YAGO dataset, I noticed that in Dataset_YG.py, you commented out the two lines of code: 104 # for i in range(start,end): 105 # year_list.append(i) I also think that the years between the start time and the end time should be counted in the year_list, but this will greatly increase the year_list, and it will also lead to a sharp increase in the effective facts in each year. What I want to know is, what made you add those two lines of code and then comment them out?

this setting refers to HyTE. During generating the time list, they counted the number of mentioned years by only considering the start time and the end time.

xjzhao001 commented 3 years ago

Dear soledad921：

    According to the operating instructions provided by the README.md in the ATISE , I reproduced icews14/icews05-15 and found that the results are very different from the results you reported. Here are my results:

ATISE/icews14/5000epoch Mean Rank: 450 Mean RR: 0.2869 Hit@1: 0.1712 Hit@3:  0.3294 Hit@5:  0.4193 Hit@10: 0.5356

ATISE/icews05-15/earlystop 2250epoch Mean Rank: 587 Mean RR: 0.2082 Hit@1: 0.1155 Hit@3:  0.2347 Hit@5:  0.2980 Hit@10: 0.3984

TERO/icews05-15/earlystop 4000epoch

Mean Rank: 603 Mean RR: 0.1651 Hit@1: 0.0155 Hit@3:  0.2433 Hit@5:  0.3330 Hit@10: 0.4618

TERO/icews14/5000epoch Mean Rank: 440 Mean RR: 0.2178 Hit@1: 0.0530 Hit@3:  0.3018 Hit@5:  0.4086 Hit@10: 0.5429

Please help me check where the problem is,

Best regards,Xiaojuan Zhao

soledad921 commented 3 years ago

Dear soledad921：     According to the operating instructions provided by the README.md in the ATISE , I reproduced icews14/icews05-15 and found that the results are very different from the results you reported. Here are my results: ATISE/icews14/5000epoch Mean Rank: 450 Mean RR: 0.2869 Hit@1: 0.1712 Hit@3:  0.3294 Hit@5:  0.4193 Hit@10: 0.5356 ATISE/icews05-15/earlystop 2250epoch Mean Rank: 587 Mean RR: 0.2082 Hit@1: 0.1155 Hit@3:  0.2347 Hit@5:  0.2980 Hit@10: 0.3984 TERO/icews05-15/earlystop 4000epoch Mean Rank: 603 Mean RR: 0.1651 Hit@1: 0.0155 Hit@3:  0.2433 Hit@5:  0.3330 Hit@10: 0.4618 TERO/icews14/5000epoch Mean Rank: 440 Mean RR: 0.2178 Hit@1: 0.0530 Hit@3:  0.3018 Hit@5:  0.4086 Hit@10: 0.5429 Please help me check where the problem is, Best regards,Xiaojuan Zhao

Dear Xiaojuan,

Thank you for your interest in my papers. I just reran these codes. After training TERO for 250 epochs, I got MRR of 0.55 on validation set of ICEWS14. As far as i can remember, the whole training process might need about 2000 epochs (each epoch costs 5 sec) and the final result on test set would be the same as the result reported in the paper. I ran my codes on a single RTX2080 GPU device with pytorch 1.4.0 and cuda 9.1. But I think it would be okay to use a more recent pytorch version if the versions of cuda and pytorch are compatible.

Hope this information could help you.

Best Wishes,

yunzi-94 commented 3 years ago

Dear soledad921： I meet the same question as xjzhao001. I run the codes many times for ICEWS14 and ICEWS05-15 according to your Readme.md file, but the results are far worse than you have posted in your papers, for both Atise and Tero. So I believe there is something wrong in the present version, and I hope you can update the codes.

soledad921 commented 3 years ago

Dear soledad921： I meet the same question as xjzhao001. I run the codes many times for ICEWS14 and ICEWS05-15 according to your Readme.md file, but the results are far worse than you have posted in your papers, for both Atise and Tero. So I believe there is something wrong in the present version, and I hope you can update the codes.

Hi, Yunzi. Thank you for your concern. I also received an email from another guy saying that he met the same issue as you and xjzhao at first. But later he got the same results as ours later. Below are his emails: ... Dear Mr. Xu: I am a master student, and my research direction is knowledge graph complement . I am very interested in your research and thank you very much for your code. But after running the source code you shared, I found that the results of the ATiSE model and thethe TeRo model on the two data sets CEWS14 and ICEWS05-15 are very different from the results in the paper. For example, using the hyperparameter value you set, I only get 0.21 and 0.55 of MRR and HITS10 on CEWS14 dataset. Could you help me analyze the possible reasons for this result? For example, the hardware configuration or package version you use, and some points of your runtime, etc. Thank you very much! best wishes

... .... Dear Mr. Xu:

Sorry for my late reply and thank you for your concern. I ran the code on another machine and got the same results as reported in your paper. After that, I created a new python environment in anaconda of my server and also got the same results as reported in your paper.

... So I think the issue is from the implementation enviroment, not the code.

yunzi-94 commented 3 years ago

Thank you for your reply, I will check the environment and retry it. 

soledad921 commented 3 years ago

Thank you for your reply, I will check the environment and retry it. 

I also send a new email to Mr. Zhang who successfully reproduced the results as mentioned. Hope he could tell us the difference between the two machines where he got different results or if the new python enviroment was the factor to make the difference? Once he replies to me, I will upload his reply. Hope it can be helpful to you.

yunzi-94 commented 3 years ago

Thanks for reminding me of the environment. I reproduced the results successfully. The main factor is the version of pytorch. 

soledad921 commented 3 years ago

Thanks for reminding me of the environment. I reproduced the results successfully. The main factor is the version of pytorch.  … ---Original--- From: "soledad921"<notifications@github.com> Date: Tue, Dec 8, 2020 09:46 AM To: "soledad921/ATISE"<ATISE@noreply.github.com>; Cc: "yunzi-nudt"<616973073@qq.com>;"Comment"<comment@noreply.github.com>; Subject: Re: [soledad921/ATISE] About year_list (#2) Thank you for your reply, I will check the environment and retry it.  I also send a new email to Mr. Zhang who successfully reproduced the results as mentioned. Hope he could tell us the difference between the two machines where he got different results or if the new python enviroment was the factor to make the difference? Once he replies to me, I will upload his reply. Hope it can be helpful to you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

Hi, Yunzi. Which versions of pytorch and cuda did you use at first and then?

yunzi-94 commented 3 years ago

At first, I tried the pytorch of 1.3.1, but the results were too bad.  Then I noticed you had modified the Readme.md file several days ago, and that you had added something about version control. So, I tried to rerun the codes with pytroch==1.4.0, and the results were as good as you posted in your paper.  The cuda version I used was 10.2.

yunzi-94 commented 3 years ago

Thanks again to your reply that really helps me a lot.