Seeking clarity on how edges are expected to be used in Link Prediction tasks

[zjost]

Hello team. Thanks for the wonderful work and project. I am wondering how exactly we are expected to handle edge masking in Link Prediction tasks. I've seen a number of issues (e.g., #213, #27) that have similar confusion. As I understand it, there are two separate types of "information leak" that could exist:

• Type A: Validation/Test edges included in the graph used for message passing. This will clearly tend to make embeddings of true source/destination pairs closer together, particularly when the graph is sparse, since the message passing operation will include the embedding of the connected node. This is likely to be amplified when there are learnable node embeddings, as in ogbl-ddi.
• Type B: Reverse edges included in the graph used for message passing. Regardless of train/val/test, there are similar concerns of including the messages from source/destination pairs when calculating their embeddings.

My questions are as follows as it relates to expectations and implementations used for the leaderboard:

1. During training, are validation/test edges included in the graph used for message passing? (E.g., adj_t here)
2. During inference of any kind, are the edges that are being inferred about somehow excluded from message passing? I know DGL provides some functionality in this direction using the exclude argument in the EdgeLoader

Ultimately, I want to make sure I'm making a fair comparison to the leaderboard, but I also wonder about the "right" way to measure performance on this task, regardless of decisions made for the leaderboard.

As a final request, perhaps the website can document these choices and expectations somewhere? Thanks!

[weihua916]

Great question. The rule to use val/test labels (edges) are clearly stated here. In short, you should not use val/test edges as message passing nor computing your loss.

[zjost]

Thank you for the response. As I understand it, you're saying the "Type A" leaks described above are avoided by excluding val/test edges from ALL message passing operations. And I suppose whether or not you include the training edges for message passing is a choice of the modeler and if it's a problem, the leaderboard will reflect that. Is that a fair assessment?

I'll close the issue to tidy up, but please feel free to correct anything I've misstated for posterity.

[weihua916]

Yes your understanding is correct

[Barcavin]

Just want to follow up on this question.

The example code uses the entire edge sets, including all training,validation and test, for the message passing purpose. adj_t here. but in leaderboard comparison, we shouldn't use it, because it leaks the edge information.

I checked the ogbl-ddi's leaderboard and the code for PLNLP and SEAL methods. it seems they both use adj_t for message passing. But it is not acceptable in this task setting, right?

[weihua916]

adj_t only contains the training edges. Message passing should never use val/test edges.