Neural ODE for Diffusion

[barkincavdaroglu]

Many graph and node-level problems are fundamentally about diffusion. Information diffuses from one node to another. In this way, solutions to these problems can be modeled after heat kernel, which satisfies heat (differential) equation, which is about diffusion of heat in surrounding environment. Heat kernel is also used to analyze various properties of random walks on graphs. With this in mind, a Neural ODE model on graphs can result in better performance for our problem than GNN + RNN/Transformer.

https://blog.twitter.com/engineering/en_us/topics/insights/2021/graph-neural-networks-as-neural-diffusion-pdes

[uhuruhashimoto]

With this in mind, a Neural ODE model on graphs can result in better performance for our problem than GNN + RNN/Transformer.

Mind adding some further justification here -- what properties of link prediction make it best modeled as an ODE/diffusion problem? Would we run into any problems regarding the fact that we're modeling a MANET, while Twitter in the blog post is modeling a social media network -- do these two graphs behave differently in any way?

[barkincavdaroglu]

With this in mind, a Neural ODE model on graphs can result in better performance for our problem than GNN + RNN/Transformer.

Mind adding some further justification here -- what properties of link prediction make it best modeled as an ODE/diffusion problem? Would we run into any problems regarding the fact that we're modeling a MANET, while Twitter in the blog post is modeling a social media network -- do these two graphs behave differently in any way?

Great question! Will share more resources soon. But for now: what GNN does is already very similar to diffusion, since information from neighboring nodes is aggregated to root node. The idea is that Neural ODEs can do it better. Whether this can be directly applied to MANETs is uncertain, and we probably won't know about it until much later (since there is no dataset available for MANETs). I think in some indirect way it does, though. Because as information from nodes diffuse, we also encode information about the SNR kernel. Although any model we choose will not be optimal for MANETs (since the weight of the edge is influenced by things we know nothing about such as the actual topology and environmental factors present in the area), the model should in theory be able to learn an approximate kernel for these causes, if that makes sense.

[uhuruhashimoto]

That was a great explanation! Sounds like a really cool idea.

we probably won't know about it until much later (since there is no dataset available for MANETs)

You mentioned there is no dataset available for MANETS -- do you mean data to train/test on? If so, what are our limitations on testing this model? (Are we limited to generating our own datasets)?

[barkincavdaroglu]

You mentioned there is no dataset available for MANETS -- do you mean data to train/test on?

Exactly! There are very few (most of which are bad quality) datasets for communication networks, but their nodes are stationary so they are a bit different. The nodes in the dataset we are currently using are also stationary, but it's a good quality and has sufficient amount of samples to make strong assumptions about the quality of the model since we are still predicting edges.

[barkincavdaroglu]

In production, when users save their data, we won't have good amount for maybe a year or two, which is not ideal. A remedy could be to train a GNN model at the global level (using data from all areas) and training Neural ODE at the local level (to approximate the kernel for the specific area).

[barkincavdaroglu]

There have been cases where people use simulation data to train/test their models' performance. So that's also an option.