search

TimDettmers / ConvE

Convolutional 2D Knowledge Graph Embeddings resources
MIT License
675 stars 163 forks source link

Model not learning anything #37

Closed aayushee closed 5 years ago

aayushee commented 5 years ago

Hi I have a Movie KG for which I am trying to train ConvE model. It has 6 relation types, around 38k entities and 100k triples. I trained ConvE model for 500 epochs and 300 dimensional vectors keeping rest of the parameters same, but the MRR and Hit@10 values never go beyond 0.2. Each epoch doesn't take more than a minute to finish. Also, the values remain same from 20 epochs of training to 500 epochs. The training loss is also 0.0003 and hardly changes. Any idea how many epochs model should take to learn the embeddings?

Following is a snapshot of the output I get:

2019-01-14 23:43:38.559798 (INFO): ######################################## 2019-01-14 23:43:38.559814 (INFO): COMPLETED EPOCH: 500
2019-01-14 23:43:38.559827 (INFO): train Loss: 0.00030636 99% CI: (0.0003061, 0.00030663), n=61 2019-01-14 23:43:38.559837 (INFO): ######################################## 2019-01-14 23:43:38.559848 (INFO):

saving to saved_models/D1_ConvE_128_500.model 2019-01-14 23:43:38.705671 (INFO): 2019-01-14 23:43:38.705735 (INFO): -------------------------------------------------- 2019-01-14 23:43:38.705770 (INFO): dev_evaluation 2019-01-14 23:43:38.706107 (INFO): -------------------------------------------------- 2019-01-14 23:43:38.706128 (INFO): 2019-01-14 23:45:19.866323 (INFO): Hits left @1: 0.3492588141025641 2019-01-14 23:45:19.866904 (INFO): Hits right @1: 0.05999599358974359 2019-01-14 23:45:19.868125 (INFO): Hits @1: 0.20462740384615385 2019-01-14 23:45:19.868674 (INFO): Hits left @2: 0.34935897435897434 2019-01-14 23:45:19.869216 (INFO): Hits right @2: 0.060196314102564104 2019-01-14 23:45:19.870324 (INFO): Hits @2: 0.20477764423076922 2019-01-14 23:45:19.870869 (INFO): Hits left @3: 0.34935897435897434 2019-01-14 23:45:19.871412 (INFO): Hits right @3: 0.060196314102564104 2019-01-14 23:45:19.872464 (INFO): Hits @3: 0.20477764423076922 2019-01-14 23:45:19.873017 (INFO): Hits left @4: 0.34935897435897434 2019-01-14 23:45:19.873587 (INFO): Hits right @4: 0.060196314102564104 2019-01-14 23:45:19.874643 (INFO): Hits @4: 0.20477764423076922 2019-01-14 23:45:19.875188 (INFO): Hits left @5: 0.34935897435897434 2019-01-14 23:45:19.875729 (INFO): Hits right @5: 0.060196314102564104 2019-01-14 23:45:19.876787 (INFO): Hits @5: 0.20477764423076922 2019-01-14 23:45:19.877350 (INFO): Hits left @6: 0.34935897435897434 2019-01-14 23:45:19.877890 (INFO): Hits right @6: 0.06029647435897436 2019-01-14 23:45:19.878956 (INFO): Hits @6: 0.20482772435897437 2019-01-14 23:45:19.879499 (INFO): Hits left @7: 0.34935897435897434 2019-01-14 23:45:19.880037 (INFO): Hits right @7: 0.060396634615384616 2019-01-14 23:45:19.881873 (INFO): Hits @7: 0.2048778044871795 2019-01-14 23:45:19.882422 (INFO): Hits left @8: 0.34935897435897434 2019-01-14 23:45:19.882968 (INFO): Hits right @8: 0.060396634615384616 2019-01-14 23:45:19.884053 (INFO): Hits @8: 0.2048778044871795 2019-01-14 23:45:19.884598 (INFO): Hits left @9: 0.34935897435897434 2019-01-14 23:45:19.885140 (INFO): Hits right @9: 0.060396634615384616 2019-01-14 23:45:19.886244 (INFO): Hits @9: 0.2048778044871795 2019-01-14 23:45:19.886789 (INFO): Hits left @10: 0.34935897435897434 2019-01-14 23:45:19.887327 (INFO): Hits right @10: 0.060396634615384616 2019-01-14 23:45:19.888419 (INFO): Hits @10: 0.2048778044871795 2019-01-14 23:45:19.889793 (INFO): Mean rank left: 12504.717748397436 2019-01-14 23:45:19.891131 (INFO): Mean rank right: 16421.001302083332 2019-01-14 23:45:19.893638 (INFO): Mean rank: 14462.859525240385 2019-01-14 23:45:19.894989 (INFO): Mean reciprocal rank left: 0.3493859660704812 2019-01-14 23:45:19.896470 (INFO): Mean reciprocal rank right: 0.06042333463160364 2019-01-14 23:45:19.899066 (INFO): Mean reciprocal rank: 0.20490465035104244

TimDettmers commented 5 years ago

Might it be that the model is actually training correctly? How well connected are most entities in the graph? Are there entities which are very well connected?

You can check this by calculating the connections for each entity and for each relation and then you calculate a histogram of that. If most entities have 1-2 connections then its too difficult to predict anything useful — if the histogram shows this, then the cause might be the data and not the model.

Could you check this please?

aayushee commented 5 years ago

Hi I checked the statistics in my KG and it seems around 15k out of 38k entities have 1 or 2 connections only. So the KG isn't very well connected. I will probably find some other way to calculate the embeddings. Thanks for your help!