search

CarloLucibello / GraphNeuralNetworks.jl

Graph Neural Networks in Julia
https://carlolucibello.github.io/GraphNeuralNetworks.jl/dev/
MIT License
214 stars 47 forks source link

Overflows in GATConv and GATv2Conv #246

Closed graidl closed 1 year ago

graidl commented 1 year ago

In GATConv an GATv2Conv the softmax calculation for determining the attention coefficients can too easily lead to overflows, e.g., already with input values of ~100. It should be realized in a numerically more stable way, e.g ,. as in AGNNConv by using softmax_edge_neighbors. Most importantly, a robust softmax implementation subtracts the maximum before exponentiating.

graidl commented 1 year ago

I have just seen that this issue is currently addressed in PR #244.