Reproducing MagNet's Results

[ClaudMor]

Dear authors,

First, thank you very much for this package.

I've tried reproducing MagNet's Link Prediction results on Cora for the Existence sub-task using this package's models and splitting functions, but I keep consistently getting around 78% accuracy on the test set compared to around 82% as reported by the paper.

Below, you may find an MWE that attains 78% accuracy on the test set: I would ask if you could kindly point to me how it could be modified to get the same performance as in the paper.

I wrote the MWE below by applying the following modifications to the Directed Case Study reported in your documentation:

1. Set all model's and split's parameters to those found for the Existence sub-task in the Arxiv version of the paper;
2. Added the early stopping functionality with 3000 maximum epochs and patience=500, as reported in the Arxiv version.

import numpy as np
from sklearn.metrics import accuracy_score
import torch

from torch_geometric_signed_directed.utils import  link_class_split, in_out_degree
from torch_geometric_signed_directed.nn.directed import MagNet_link_prediction 
from torch_geometric_signed_directed.data import  load_directed_real_data
from torch_geometric import seed_everything

# Set the seed
seed = 12345
seed_everything(seed) 
# Set the device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Load data as specified by the paper
data = load_directed_real_data(dataset='cora_ml', root="../data/pygsd/").to(device) 
link_data = link_class_split(data, prob_val=0.05, prob_test=0.15, task = 'existence', device=device, splits = 10, seed = seed)

# Load the model with parameters as specified in the paper (?)
model = MagNet_link_prediction(q=0.05, K=1, num_features=2, dropout = 0.5, hidden=16, label_dim=2).to(device)
criterion = torch.nn.NLLLoss()

def train(X_real, X_img, y, edge_index,
edge_weight, query_edges):
    model.train()
    out = model(X_real, X_img, edge_index=edge_index,
                    query_edges=query_edges,
                    edge_weight=edge_weight)
    loss = criterion(out, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    train_acc = accuracy_score(y.cpu(),
    out.max(dim=1)[1].cpu())
    return loss.detach().item(), train_acc

def test(X_real, X_img, y, edge_index, edge_weight,
query_edges):
    model.eval()
    with torch.no_grad():
        out = model(X_real, X_img, edge_index=edge_index,
                    query_edges=query_edges,
                    edge_weight=edge_weight)
    test_acc = accuracy_score(y.cpu(),
    out.max(dim=1)[1].cpu())
    return test_acc

test_accs = []
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0.0005)
# Save optimizer and model's initial states
initial_model_state_dict = model.state_dict()
initial_optimizer_state_dict = optimizer.state_dict()

for split in list(link_data.keys()):
    # Load optimizer's and model's initial states so that they are the same for every split
    model.load_state_dict(initial_model_state_dict)
    optimizer.load_state_dict(initial_optimizer_state_dict)
    edge_index = link_data[split]['graph'] 
    edge_weight = link_data[split]['weights']
    query_edges = link_data[split]['train']['edges']
    y = link_data[split]['train']['label']
    X_real = in_out_degree(edge_index,
    size=len(data.x)).to(device)
    X_img = X_real.clone()
    query_val_edges = link_data[split]['val']['edges']
    y_val = link_data[split]['val']['label']

    best_val_acc = 0.0
    best_num_epoch = 0 
    val_acc_consecutive_violations = 0
    for epoch in range(3000):
        train_loss, train_acc = train(X_real,
        X_img, y, edge_index, edge_weight, query_edges)
        val_acc = test(X_real, X_img, y_val,
        edge_index, edge_weight, query_val_edges)
        # print(f'Split: {split:02d}, Epoch: {epoch:03d}, \
        # Train_Loss: {train_loss:.4f}, Train_Acc: \
        # {train_acc:.4f}, Val_Acc: {val_acc:.4f}')
        # Early stopping
        if val_acc <= best_val_acc:
            val_acc_consecutive_violations += 1
        elif val_acc > best_val_acc:
            best_val_acc = val_acc
            best_num_epoch = epoch
            val_acc_consecutive_violations = 0

        if val_acc_consecutive_violations > 500:
            break
    print(f"split = {split}, best_num_epoch = {best_num_epoch}, retraining...")

    # Reset optimizer and model prior to re-training
    model.load_state_dict(initial_model_state_dict)
    optimizer.load_state_dict(initial_optimizer_state_dict)

    # Re-train for the optimal number of epochs
    for epoch in range(best_num_epoch):
        train_loss, train_acc = train(X_real,
        X_img, y, edge_index, edge_weight, query_edges)

    query_test_edges = link_data[split]['test']['edges']
    y_test = link_data[split]['test']['label']
    test_acc = test(X_real, X_img, y_test, edge_index,
    edge_weight, query_test_edges)
    test_accs.append(test_acc)
    print(f'Split: {split:02d}, Test_Acc: {test_acc:.4f}')
    # No need to reset parameters since we load the states
    # model.reset_parameters()

# Print performances
print(f"{np.mean(test_accs)} +- {np.std(test_accs)}")

0.7803090332805072 +- 0.007189083720928716

How may I get 0.82 as in the paper?

@XitongSystem might also be interested as the first author of MagNet.

Thank you very much, any advice is greatly appreciated!

[XitongSystem]

Hi ClaudMor,

Thank you for the question! The implementation of the random split of edges is different.

For the original MagNet implementation here: https://github.com/matthew-hirn/magnet/blob/main/src/utils/edge_data.py#L162, we used the split based on the EdgeSplitter from stellargraph.data. Also, the random seed for sampling edges for each fold is different.

For the implementation of this package, to make the installation dependency simpler, we implemented the split based on networkx and PyG. And we use only one random seed to generate edges for all train/val folds.

To fully reproduce the results in the MagNet paper, we recommend using the split in the MagNet paper.

Hope this helps!

[ClaudMor]

Hi @XitongSystem,

It definitely helped!

Thank you very much!