search

Accenture / AmpliGraph

Python library for Representation Learning on Knowledge Graphs https://docs.ampligraph.org
Apache License 2.0
2.14k stars 251 forks source link

entities_subset in evaluate_performance for large graphs #231

Closed kalinin-sanja closed 3 years ago

kalinin-sanja commented 3 years ago

It looks like in large graph mode evaluate_performance doesn't use entities_subset. Is that true?

sumitpai commented 3 years ago

@kalinin-sanja entities_subset should work in large graph mode as well. Let me know if there are any issues with this

kalinin-sanja commented 3 years ago

@kalinin-sanja entities_subset should work in large graph mode as well. Let me know if there are any issues with this

I've reproduced this problem on the public dataset:

import numpy as np
from ampligraph.datasets import load_wn18
from ampligraph.latent_features import ComplEx, set_entity_threshold
from ampligraph.evaluation import evaluate_performance, mrr_score, hits_at_n_score, mr_score

X = load_wn18()

model = ComplEx(batches_count=10, seed=0, epochs=2, k=150, eta=10,
                optimizer='adam', optimizer_params={'lr': 1e-3},
                loss='pairwise', loss_params={'margin': 0.5},
                regularizer='LP', regularizer_params={'p': 2, 'lambda': 1e-5},
                verbose=True)

model.fit(X['train'])

_filter = np.concatenate((X['train'], X['valid'], X['test']))

all_nodes = set()
for split in ['train', 'valid', 'test']:
    all_nodes = set.union(all_nodes, set(X[split][:, 0]), set(X[split][:, 2]))
print(f'Total count of nodes: {len(all_nodes):,}')

# We will corrupt tails of specific relation with fix set of entities.
entities_subset = np.random.choice(list(all_nodes), 100, replace=False)
selected_triples = X['test'][X['test'][:, 1] == '_hyponym']
selected_nodes = set.union(set(selected_triples[:, 0]), set(selected_triples[:, 2]))

print(f'Evaluating on {len(selected_triples):,} triples, which contain {len(selected_nodes):,} entities...')
print(f'Use {len(entities_subset):,} entities for corruption of tails.')

set_entity_threshold(1)

ranks = evaluate_performance(selected_triples,
                             model=model,
                             filter_triples=_filter,
                             corrupt_side='o',
                             use_default_protocol=False,
                             entities_subset=entities_subset,
                             verbose=True)
n = 10
print(f'Ranks of first {n} triples: ', ranks[0:n])
print(f'MR: {mr_score(ranks)}')
print(f'MRR: {mrr_score(ranks)}')
print(f'Hit@1: {hits_at_n_score(ranks, n=1)}')
print(f'Hit@3: {hits_at_n_score(ranks, n=3)}')
print(f'Hit@10: {hits_at_n_score(ranks, n=10)}')
kalinin-sanja commented 3 years ago

Output:

2021-04-21 19:08:00.479004: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-21 19:08:00.488780: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2099900000 Hz
2021-04-21 19:08:00.492982: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558633b457d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-21 19:08:00.493027: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-04-21 19:08:00.497597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-04-21 19:08:00.789989: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5586341f3130 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-21 19:08:00.790048: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla P100-SXM2-16GB, Compute Capability 6.0
2021-04-21 19:08:00.790065: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Tesla P100-SXM2-16GB, Compute Capability 6.0
2021-04-21 19:08:00.793114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla P100-SXM2-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:06:00.0
2021-04-21 19:08:00.795267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
name: Tesla P100-SXM2-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:85:00.0
2021-04-21 19:08:00.796446: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-21 19:08:00.799503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-04-21 19:08:00.802094: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-04-21 19:08:00.803314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-04-21 19:08:00.806997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-04-21 19:08:00.809727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-04-21 19:08:00.817278: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-21 19:08:00.825673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1
2021-04-21 19:08:00.825752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-21 19:08:00.830249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-21 19:08:00.830281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 1 
2021-04-21 19:08:00.830308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N Y 
2021-04-21 19:08:00.830320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 1:   Y N 
2021-04-21 19:08:00.836586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14081 MB memory) -> physical GPU (device: 0, name: Tesla P100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2021-04-21 19:08:00.839196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9483 MB memory) -> physical GPU (device: 1, name: Tesla P100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 6.0)
Average Loss:   0.499194: 100%|████████████████| 2/2 [00:01<00:00,  1.10epoch/s]
Total count of nodes: 40,943
Evaluating on 1,153 triples, which contain 2,033 entities...
Use 100 entities for corruption of tails.
WARNING - Your graph has a large number of distinct entities. Found 40943 distinct entities
WARNING - Changing the variable loading strategy to use lazy loading of variables...
WARNING - Evaluation would take longer than usual.
***/anaconda3/envs/tf1_cuda/lib/python3.6/site-packages/ampligraph/latent_features/models/EmbeddingModel.py:1140: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if corruption_entities == 'all':
2021-04-21 19:08:07.566083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla P100-SXM2-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:06:00.0
2021-04-21 19:08:07.567303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
name: Tesla P100-SXM2-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:85:00.0
2021-04-21 19:08:07.567397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-21 19:08:07.567420: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-04-21 19:08:07.567439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-04-21 19:08:07.567461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-04-21 19:08:07.567480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-04-21 19:08:07.567498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-04-21 19:08:07.567519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-21 19:08:07.572063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1
2021-04-21 19:08:07.572127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-21 19:08:07.572139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 1 
2021-04-21 19:08:07.572147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N Y 
2021-04-21 19:08:07.572155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 1:   Y N 
2021-04-21 19:08:07.575811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14081 MB memory) -> physical GPU (device: 0, name: Tesla P100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2021-04-21 19:08:07.576977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9483 MB memory) -> physical GPU (device: 1, name: Tesla P100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 6.0)
  0%|                                                  | 0/1153 [00:00<?, ?it/s]
***/anaconda3/envs/tf1_cuda/lib/python3.6/site-packages/ampligraph/latent_features/models/EmbeddingModel.py:1091: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if corruption_entities == 'all':
100%|███████████████████████████████████████| 1153/1153 [06:16<00:00,  3.06it/s]
Ranks of first 10 triples:  [40907 40927    13     6 40916     6    14 40912    31     1]
MR: 3238.659150043365
MRR: 0.3486335751733688
Hit@1: 0.212489158716392
Hit@3: 0.4006938421509107
Hit@10: 0.6183868169991327
kalinin-sanja commented 3 years ago

One can note a strange warning:

/anaconda3/envs/tf1_cuda/lib/python3.6/site-packages/ampligraph/latent_features/models/EmbeddingModel.py:1091: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if corruption_entities == 'all':
kalinin-sanja commented 3 years ago 
tensorflow-gpu==1.15.3
kalinin-sanja commented 3 years ago

Skimming reading: image image

sumitpai commented 3 years ago

you can ignore the warning, it is because the corruption_entities is now a numpy array. Consider the example below: 

import numpy as np
x = np.array([1,2,3], dtype=np.float32)
# x = 'all'
if x == 'all':
    print(x)
elif isinstance(x, np.ndarray):
    print('x is an array')
else:
    print('else')

We will make sure that, in the future, this warning doesn't appear.

However, in your case, the entities subset works correctly and you can see the resulting mrr and ranks.

kalinin-sanja commented 3 years ago

Maybe I misunderstood the meaning of entities_subset. Let's assume we want to predict a movie of actor, i.e., <ACTOR, ACT_IN, ?>. We have a subset of movies: [A, B, C, D, E]. So, we want to generate five corruptions for every test triple. Obviously, rank could not be more than len(entities_subset) + 1, could it?

sumitpai commented 3 years ago

Okay now I understand what you mean. I guess there is some bug. I will have a look at it.

yeah entities_subset is supposed to corrupt only from the set you specify and give a rank between [1, len(entities_subset) + 1].

Thanks for identifying this issue. I will take up this issue on priority.

The issue seems to exist in large graph mode only.

sumitpai commented 3 years ago

Fixed on feature/231

sumitpai commented 3 years ago

merged with develop

kalinin-sanja commented 3 years ago

merged with develop

So, when we should wait for the new release? Will it use TF2?

sumitpai commented 3 years ago

By next week we will push it to master, until then you can pip install from the develop branch. It will still be using tf 1.x.

We are working on a tf 2.x release, and it will be out soon.