search

divelab / GOOD

GOOD: A Graph Out-of-Distribution Benchmark [NeurIPS 2022 Datasets and Benchmarks]
https://good.readthedocs.io/
GNU General Public License v3.0
180 stars 19 forks source link

How to fix random seeds？ #27

Closed bruno686 closed 4 months ago

bruno686 commented 5 months ago

Every time I run it out the results are different. Tried setting random seeds in both the args, and yaml files to no avail...

CM-BF commented 5 months ago

Sorry for the inconvenience. It sounds weird. Random seeds are reset in many places including the dataloader. Would you mind to 1) compare the model weights before training and 2) compare the first batch data between two runs? Once I get a moment, I will look into it.

bruno686 commented 5 months ago

Hi Shurui, after much review, I found that the initial data is consistent for each run, but the initial weights are not. I hadn't noticed this problem before on the CBAS dataset(Model: ERM), and only on closer inspection today did I realise that on the CBAS dataset there are only very few values that are slightly different each run. But on WebKB(Model: ERM) this problem is magnified a lot, and there is almost always a large inconsistency each run. (You can try running just one epoch to quickly check the results)