rusty1s
commented
11 months ago Sorry for late reply. I think these issues are super hard to track down. From my personal experience, I don't see this at all on my use-cases. A few things to try:
- What is the GNN operator you are using? If you are using sum aggregation, you usually have to apply normalization in order to ensure that features are within reasonable range and to prevent gradients to explode.
- What normalization layer do you apply in the first place? I suggest to try
LayerNorm(mode="node"), which I have the best experience made with. - This can also be a sign that your LR is too high, so you can explore LR schedulers and/or smaller learning rates.
🐛 Describe the bug
I'm currently having problems similar to those detailed in issue #1478. Comments on that issue, as well as the linked Stats SE post, mostly depict the occurrence of loss spikes as instabilities that are more or less unavoidable when using GNNs. However, the latest update to either is by now over 3 years old (not counting the tangentially related issue #2004; but that's just slightly more recent, anyway). Furthermore, it was also mentioned that while GNNs encounter some instabilities, there has been "no thorough study about this".
Since I'm also seeing similarly (as far as my expertise goes unexplainable) loss spikes in the project I'm currently working on, I'm curious as to whether there have been any updates concerning this behaviour in the meantime, that didn't make it into the old issue(s). I myself couldn't find any studies on the problem, but I'm hoping someone among the maintainers of PyG may have more insight. More specifically, I ultimately would like to know if this is just behaviour I'll have to live with, or if the bug has a potential solve by now.
As for my own testing so far: since parts of the discussion suggest the optimizer may be at fault, I have tried using both the Adamax and SGD optimizers, but neither option eliminated the loss spikes. I have not tried the mentioned gradient clipping, but the previous discussion doesn't suggest that it may be of much help, anyway. Side note: I'm aware Python 3.7 is not exactly the latest version of Python, but it's a limitation of the environment I'm working in that I cannot currently change; the same is true for being restricted to CPU-based training.
For illustrative purposes, here's an exemplary loss graph visualizing the problem:
Environment
conda,pip, source): piptorch-scatter): pyg-lib 0.1.0+pt113cpu torch-cluster 1.6.0+pt113cpu torch-scatter 2.0.9 torch-sparse 0.6.16+pt113cpu torch-spline-conv 1.2.1+pt113cpu