Config for Freebase knowledge graph when training with different number of partitions

[DinikaSen]

Hi, Can I know the values used as learning rate, batch size etc. when training Freebase knowledge graph with different number of partitions? If you have performed distributed training for other graph datasets too, can I know the hyperparameters with the best performance for each number of partitions?

[lw]

Hi, sorry for the very late reply. I didn't run the experiments myself so I don't have the configs readily available. Perhaps @adamlerer or @ledw have them? Did you try just running the config we provide with a higher number of partitions? Do you get worse results than the ones published in the paper?

[rom1504]

Hello @lerks , I have a similar questions about the parameters you used for freebase. (thanks for the previous answer on importing data efficiently, I did manage to generate the dataset quite efficiency using spark, I'll provide more details about that later) For freebase, did you use a config similar to https://github.com/facebookresearch/PyTorch-BigGraph/blob/master/torchbiggraph/examples/configs/fb15k_config.py for training or more like the one for freebase https://torchbiggraph.readthedocs.io/en/latest/pretrained_embeddings.html ? I'm particularly interested by the parameters that influence training speed like embeddings size, number of epochs and parameters related to batches

[rom1504]

I was able to reproduce the results of the paper on the big freebase with this config :

def get_torchbiggraph_config():

    config = dict(
        # I/O data
        entity_path="data/freebase/large",
        edge_paths=[
            "data/freebase/large/edges"
            # "data/freebase/large/graph_v0_valid-partitioned",
            # "data/freebase/large/graph_v0_test-partitioned",
        ],
        checkpoint_path="model/freebase/large",

        # Graph structure
        entities={
            'all': {'num_partitions': 4},
        },
        relations=[{
            'name': 'all_edges',
            'lhs': 'all',
            'rhs': 'all',
            'operator': 'complex_diagonal',
        }],
        dynamic_relations=True,

        # Scoring model
        dimension=100,
        global_emb=False,
        comparator='dot',

        # Training
        num_epochs=3,
        num_edge_chunks=10,
        batch_size=10000,
        num_batch_negs=500,
        num_uniform_negs=500,
        loss_fn='softmax',
        relation_lr=0.01,
        lr=0.1,

        # Evaluation during training
        eval_fraction=0.001,
        eval_num_batch_negs=10000,
        eval_num_uniform_negs=0,

        # Misc
        background_io=False,
        verbose=1,
    )

    return config

it took about 20h to train in my case. Metrics (unfiltered) :

2020-01-29 17:05:47,758   [Evaluator] Stats: pos_rank:  203.031 , mrr:  0.278492 , r1:  0.252063 , r10:  0.319776 , r50:  0.401484 , auc:  0.645018 , count:  263608638