search

snap-stanford / ogb

Benchmark datasets, data loaders, and evaluators for graph machine learning
https://ogb.stanford.edu
MIT License
1.89k stars 397 forks source link

ogbg-code2 returns F1=0. all the time #290

Closed JeanKaddour closed 2 years ago

JeanKaddour commented 2 years ago

Hi OGB team, first of all, thank you very much for this great resource and your work!

When I run examples/graphproppred/code2/main_pyg.py, I get F1=0. all the time, for train, valid, and test set.

In the first screenshot, I'm showing the results for the first three epochs only. This is what I get by just running your script and not touching anything, not even the logging. In the second screenshot, you can see that the training loss doesn't improve much even over the default 25 epochs. Again, the final F1 scores here are all 0.

image image

weihua916 commented 2 years ago

Hi! This might be related to https://github.com/snap-stanford/ogb/issues/260 Can you reprocess the dataset after you install/update PyG?

JeanKaddour commented 2 years ago

Thanks for your response! I just re-installed PyG from master, and ran the script again after deleting the folder dataset entirely (so the data was downloaded and processed again). It didn't help. I still get F1 scores either being exactly zero 0 or almost zero. image

weihua916 commented 2 years ago

Hi! I used Pytorch version 1.10.0 and was able to reproduce the same issue. It turns out that the root cause is Pytorch's bug on CrossEntropyLoss (see here)!

The issue has been resolved in 1.10.1. Once I updated the Pytorch version, I obtained non-zero F1 scores:

[10030 rows x 2 columns]

Params: 13841815

=====Epoch 1 Training... Iteration: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3188/3188 [05:33<00:00, 9.56it/s] Average training loss: 3.1366342785516976 Evaluating... Iteration: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3188/3188 [03:02<00:00, 17.43it/s] Iteration: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 179/179 [00:13<00:00, 13.02it/s] Iteration: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 172/172 [00:11<00:00, 14.94it/s] {'Train': {'precision': 0.09623143681654477, 'recall': 0.05144160132413116, 'F1': 0.06412746542938594}, 'Validation': {'precision': 0.08581320944909497, 'recall': 0.04399707889759167, 'F1': 0.05549395431281744}, 'Test': {'precision': 0.08448150173136505, 'recall': 0.043179188326809975, 'F1': 0.05463354811359185}}

JeanKaddour commented 2 years ago

Thank you so much, upgrading to 1.10.1 indeed solved it!