search

amzn / ss-aga-kgc

Apache License 2.0
31 stars 9 forks source link

When training "alignment model", the result is poor #11

Open jiazhaojun opened 2 years ago

jiazhaojun commented 2 years ago

Hello, thank you for opening this code, but I found in the process of training the model that the verification loss of "alignment model" is very high and the result is very poor from the first epoch. The situation of the 0 epoch is very different. Do you know the specific reason。The specific results are as follows: I am a graduate student at school. I look forward to your reply. Thank you

2022-09-12 05:52:41,090 INFO Align ja en Epoch 0 [Train Align Loss 0.577126| 2022-09-12 05:52:44,719 INFO Align ja en Epoch 1 [Train Align Loss 0.294240| 2022-09-12 05:52:50,049 INFO Align es en Epoch 0 [Train Align Loss 0.224772| 2022-09-12 05:52:55,915 INFO Align es en Epoch 1 [Train Align Loss 0.179589| 2022-09-12 05:53:00,731 INFO Align ja fr Epoch 0 [Train Align Loss 0.215952| 2022-09-12 05:53:05,533 INFO Align ja fr Epoch 1 [Train Align Loss 0.201016| 2022-09-12 05:53:11,439 INFO Align en fr Epoch 0 [Train Align Loss 0.184156| 2022-09-12 05:53:17,333 INFO Align en fr Epoch 1 [Train Align Loss 0.186263| 2022-09-12 05:53:23,425 INFO Align es fr Epoch 0 [Train Align Loss 0.188306| 2022-09-12 05:53:29,524 INFO Align es fr Epoch 1 [Train Align Loss 0.187599| 2022-09-12 05:53:30,719 INFO Align el ja Epoch 0 [Train Align Loss 0.207775| 2022-09-12 05:53:31,885 INFO Align el ja Epoch 1 [Train Align Loss 0.202363| 2022-09-12 05:53:36,116 INFO Align ja es Epoch 0 [Train Align Loss 0.202225| 2022-09-12 05:53:40,436 INFO Align ja es Epoch 1 [Train Align Loss 0.191727| 2022-09-12 05:53:42,626 INFO Align el fr Epoch 0 [Train Align Loss 0.222263| 2022-09-12 05:53:44,750 INFO Align el fr Epoch 1 [Train Align Loss 0.185255| 2022-09-12 05:53:46,474 INFO Align el en Epoch 0 [Train Align Loss 0.202170| 2022-09-12 05:53:48,185 INFO Align el en Epoch 1 [Train Align Loss 0.180836| 2022-09-12 05:53:50,166 INFO Align el es Epoch 0 [Train Align Loss 0.202020| 2022-09-12 05:53:52,150 INFO Align el es Epoch 1 [Train Align Loss 0.172025| 2022-09-12 05:54:08,436 INFO KG ja Epoch 0 [Train KG Loss 0.474746| 2022-09-12 05:54:24,647 INFO KG ja Epoch 1 [Train KG Loss 0.240753| 2022-09-12 05:54:40,986 INFO KG ja Epoch 2 [Train KG Loss 0.140252| 2022-09-12 05:54:51,248 INFO KG el Epoch 0 [Train KG Loss 0.430338| 2022-09-12 05:55:01,442 INFO KG el Epoch 1 [Train KG Loss 0.187315| 2022-09-12 05:57:17,300 INFO KG en Epoch 0 [Train KG Loss 0.360577| 2022-09-12 05:59:33,198 INFO KG en Epoch 1 [Train KG Loss 0.136722| 2022-09-12 06:01:16,357 INFO KG es Epoch 0 [Train KG Loss 0.280173| 2022-09-12 06:02:59,333 INFO KG es Epoch 1 [Train KG Loss 0.107179| 2022-09-12 06:04:39,537 INFO KG fr Epoch 0 [Train KG Loss 0.252925| 2022-09-12 06:06:19,625 INFO KG fr Epoch 1 [Train KG Loss 0.107643| 2022-09-12 06:06:19,626 INFO === round 0 2022-09-12 06:06:19,626 INFO [ja] 2022-09-12 06:06:22,872 INFO Val: Hits@1 (8633 triples): 0.000811 2022-09-12 06:06:22,872 INFO Val: Hits@10 (8633 triples): 0.005444 2022-09-12 06:06:22,874 INFO Val: MRR (8633 triples): 0.003637 2022-09-12 06:06:24,395 INFO Test: Hits@1 (2162 triples): 0.001850 2022-09-12 06:06:24,395 INFO Test: Hits@10 (2162 triples): 0.006938 2022-09-12 06:06:24,396 INFO Test: MRR (2162 triples): 0.004615 2022-09-12 06:06:24,396 INFO BestVal! Epoch 0000 [Test seq] | Best mrr 0.004615| hits1 0.001850| hits10 0.006938| 2022-09-12 06:06:25,716 INFO BestTest! Epoch 0000 [Test seq] | Best mrr 0.004615| hits1 0.001850| hits10 0.006938| 2022-09-12 06:06:25,717 INFO Epoch: 1 2022-09-12 06:06:29,338 INFO Align ja en Epoch 0 [Train Align Loss 8.607937| 2022-09-12 06:06:32,889 INFO Align ja en Epoch 1 [Train Align Loss 5.972235| 2022-09-12 06:06:38,230 INFO Align es en Epoch 0 [Train Align Loss 6.809273| 2022-09-12 06:06:43,511 INFO Align es en Epoch 1 [Train Align Loss 6.584582| 2022-09-12 06:06:48,272 INFO Align ja fr Epoch 0 [Train Align Loss 6.047226| 2022-09-12 06:06:53,006 INFO Align ja fr Epoch 1 [Train Align Loss 5.838850| 2022-09-12 06:06:58,839 INFO Align en fr Epoch 0 [Train Align Loss 6.712930| 2022-09-12 06:07:04,671 INFO Align en fr Epoch 1 [Train Align Loss 6.519938| 2022-09-12 06:07:10,778 INFO Align es fr Epoch 0 [Train Align Loss 6.554777| 2022-09-12 06:07:16,835 INFO Align es fr Epoch 1 [Train Align Loss 6.365027| 2022-09-12 06:07:18,000 INFO Align el ja Epoch 0 [Train Align Loss 5.482096| 2022-09-12 06:07:19,153 INFO Align el ja Epoch 1 [Train Align Loss 5.316339| 2022-09-12 06:07:23,323 INFO Align ja es Epoch 0 [Train Align Loss 5.814239| 2022-09-12 06:07:27,476 INFO Align ja es Epoch 1 [Train Align Loss 5.632405| 2022-09-12 06:07:29,557 INFO Align el fr Epoch 0 [Train Align Loss 6.017292| 2022-09-12 06:07:31,627 INFO Align el fr Epoch 1 [Train Align Loss 5.835585| 2022-09-12 06:07:33,286 INFO Align el en Epoch 0 [Train Align Loss 6.054560| 2022-09-12 06:07:34,944 INFO Align el en Epoch 1 [Train Align Loss 5.876427| 2022-09-12 06:07:36,959 INFO Align el es Epoch 0 [Train Align Loss 5.998742| 2022-09-12 06:07:38,917 INFO Align el es Epoch 1 [Train Align Loss 5.883946| 2022-09-12 06:07:55,090 INFO KG ja Epoch 0 [Train KG Loss 0.133453| 2022-09-12 06:08:11,212 INFO KG ja Epoch 1 [Train KG Loss 0.072645| 2022-09-12 06:08:27,317 INFO KG ja Epoch 2 [Train KG Loss 0.048118| 2022-09-12 06:08:37,599 INFO KG el Epoch 0 [Train KG Loss 0.134991| 2022-09-12 06:08:47,770 INFO KG el Epoch 1 [Train KG Loss 0.078373| 2022-09-12 06:11:02,964 INFO KG en Epoch 0 [Train KG Loss 0.123032| 2022-09-12 06:13:21,543 INFO KG en Epoch 1 [Train KG Loss 0.080099| 2022-09-12 06:15:04,463 INFO KG es Epoch 0 [Train KG Loss 0.089405| 2022-09-12 06:16:47,372 INFO KG es Epoch 1 [Train KG Loss 0.066886| 2022-09-12 06:18:27,494 INFO KG fr Epoch 0 [Train KG Loss 0.096441| 2022-09-12 06:20:07,768 INFO KG fr Epoch 1 [Train KG Loss 0.069282| 2022-09-12 06:20:07,769 INFO === round 1 2022-09-12 06:20:07,769 INFO [ja] 2022-09-12 06:20:11,016 INFO Val: Hits@1 (8633 triples): 0.000695 2022-09-12 06:20:11,016 INFO Val: Hits@10 (8633 triples): 0.018881 2022-09-12 06:20:11,019 INFO Val: MRR (8633 triples): 0.008707 2022-09-12 06:20:12,544 INFO Test: Hits@1 (2162 triples): 0.000925 2022-09-12 06:20:12,544 INFO Test: Hits@10 (2162 triples): 0.019426 2022-09-12 06:20:12,544 INFO Test: MRR (2162 triples): 0.008669 2022-09-12 06:20:12,545 INFO BestVal! Epoch 0001 [Test seq] | Best mrr 0.008669| hits1 0.000925| hits10 0.019426| 2022-09-12 06:20:13,887 INFO BestTest! Epoch 0001 [Test seq] | Best mrr 0.008669| hits1 0.000925| hits10 0.019426| 2022-09-12 06:20:13,888 INFO Epoch: 2 2022-09-12 06:20:17,512 INFO Align ja en Epoch 0 [Train Align Loss 7.293987| 2022-09-12 06:20:21,065 INFO Align ja en Epoch 1 [Train Align Loss 6.933926| 2022-09-12 06:20:26,339 INFO Align es en Epoch 0 [Train Align Loss 8.175304| 2022-09-12 06:20:31,574 INFO Align es en Epoch 1 [Train Align Loss 7.902112| 2022-09-12 06:20:36,383 INFO Align ja fr Epoch 0 [Train Align Loss 6.991634| 2022-09-12 06:20:41,173 INFO Align ja fr Epoch 1 [Train Align Loss 6.720252| 2022-09-12 06:20:47,038 INFO Align en fr Epoch 0 [Train Align Loss 8.128806| 2022-09-12 06:20:52,879 INFO Align en fr Epoch 1 [Train Align Loss 7.845694| 2022-09-12 06:20:58,964 INFO Align es fr Epoch 0 [Train Align Loss 7.800725| 2022-09-12 06:21:05,067 INFO Align es fr Epoch 1 [Train Align Loss 7.538444| 2022-09-12 06:21:06,247 INFO Align el ja Epoch 0 [Train Align Loss 5.946376| 2022-09-12 06:21:07,410 INFO Align el ja Epoch 1 [Train Align Loss 5.725021| 2022-09-12 06:21:11,578 INFO Align ja es Epoch 0 [Train Align Loss 6.610154| 2022-09-12 06:21:15,736 INFO Align ja es Epoch 1 [Train Align Loss 6.362963| 2022-09-12 06:21:17,811 INFO Align el fr Epoch 0 [Train Align Loss 6.986602| 2022-09-12 06:21:19,904 INFO Align el fr Epoch 1 [Train Align Loss 6.773473| 2022-09-12 06:21:21,579 INFO Align el en Epoch 0 [Train Align Loss 7.147122| 2022-09-12 06:21:23,266 INFO Align el en Epoch 1 [Train Align Loss 6.931155| 2022-09-12 06:21:25,220 INFO Align el es Epoch 0 [Train Align Loss 6.472295| 2022-09-12 06:21:27,170 INFO Align el es Epoch 1 [Train Align Loss 6.281886| 2022-09-12 06:21:43,375 INFO KG ja Epoch 0 [Train KG Loss 0.077415| 2022-09-12 06:21:59,608 INFO KG ja Epoch 1 [Train KG Loss 0.053311|

ZijieH commented 2 years ago

Have you installed all the packages with the right version? I got the following results from the first epoch in one saved model:

2022-03-28 13:41:56 INFO === round 0 2022-03-28 13:41:56 INFO [ja] 2022-03-28 13:42:04 INFO Val: Hits@1 (8633 triples): 0.069501 2022-03-28 13:42:04 INFO Val: Hits@10 (8633 triples): 0.197730 2022-03-28 13:42:04 INFO Val: MRR (8633 triples): 0.115438 2022-03-28 13:42:09 INFO Test: Hits@1 (2162 triples): 0.077243 2022-03-28 13:42:09 INFO Test: Hits@10 (2162 triples): 0.198427 2022-03-28 13:42:09 INFO Test: MRR (2162 triples): 0.122049 2022-03-28 13:42:09 INFO Epoch 0000 [Test seq] | Best mrr 0.122049| hits1 0.077243| hits10 0.198427|

jiazhaojun commented 2 years ago

The main package versions are as follows:

torch 1.11.0

cuda 11.3

python 3.7/3.8

PIP install directly used by torch geometric and torch scatter torch spark torch cluster

There were some errors when I first ran the code, so I modified some places, such as:

 line 184 of the source\ssaga_model.py script

neg_ loss = torch. mean(neg_losses, dim=-1)

target = torch. tensor([-1], dtype=torch.long, device=self.device)

Changed to

neg_ loss = torch.unsqueeze(torch.mean(neg_losses, dim=-1), 1)target = torch.tensor([[-1]*pos_loss.size(0)], dtype=torch. long, device=self.device)

Because they and pos_loss The dimension is different, resulting in

loss = self. criterion_ Kg (pos_loss, neg_loss, target)

------------------ 原始邮件 ------------------ 发件人: "amzn/ss-aga-kgc" @.>; 发送时间: 2022年9月12日(星期一) 下午3:01 @.>; @.**@.>; 主题: Re: [amzn/ss-aga-kgc] When training "alignment model", the result is poor (Issue #11)

Have you installed all the packages with the right version? I got the following results from the first epoch in one saved model:

2022-03-28 13:41:56 INFO === round 0 2022-03-28 13:41:56 INFO [ja] 2022-03-28 13:42:04 INFO Val: @. (8633 triples): 0.069501 2022-03-28 13:42:04 INFO Val: @. (8633 triples): 0.197730 2022-03-28 13:42:04 INFO Val: MRR (8633 triples): 0.115438 2022-03-28 13:42:09 INFO Test: @. (2162 triples): 0.077243 2022-03-28 13:42:09 INFO Test: @. (2162 triples): 0.198427 2022-03-28 13:42:09 INFO Test: MRR (2162 triples): 0.122049 2022-03-28 13:42:09 INFO Epoch 0000 [Test seq] | Best mrr 0.122049| hits1 0.077243| hits10 0.198427|

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

ZijieH commented 2 years ago

Please refer to the readme file to install the corresponding version for running the code.

jiazhaojun commented 2 years ago

OK，thanks

---Original--- From: @.> Date: Mon, Sep 12, 2022 17:03 PM To: @.>; Cc: @.**@.>; Subject: Re: [amzn/ss-aga-kgc] When training "alignment model", the result ispoor (Issue #11)

Please refer to the readme file to install the corresponding version for running the code.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>