search

QueuQ / SSM

9 stars 2 forks source link

Problem of reproducing baseline experiments. #1

Open superallen13 opened 1 year ago

superallen13 commented 1 year ago

Hi QueuQ,

Thanks for your amazing work of continual graph learning. I recently tried to reproduce the Table II in your paper, but I found I only can get similar results with your other work (CGLB).

For example, in the setting of class_IL without interedges, GCLB gives AP = 3.0±0.1 in the CoraFull dataset, but in your SSM paper, the AP is 34.5±4.4. I was just wondering if you changed some experiments setting or found some better hyperparameter settings?

Cheers, Allen

QueuQ commented 1 year ago

Hi QueuQ,

Thanks for your amazing work of continual graph learning. I recently tried to reproduce the Table II in your paper, but I found I only can get similar results with your other work (CGLB).

For example, in the setting of class_IL without interedges, GCLB gives AP = 3.0±0.1 in the CoraFull dataset, but in your SSM paper, the AP is 34.5±4.4. I was just wondering if you changed some experiments setting or found some better hyperparameter settings?

Cheers, Allen

Hi Allen,

Thanks a lot for your interest in our work! The difference in the results is caused by the backbones. As explained in Section B 2) of this paper (SSM), we tested different backbones and report the best results. This is because we want to show that SSM can outperform the baselines with whatever backbone. While in the CGLB work, as explained in Section 4, GCN is used for all baselines for a fair comparison.

ZhangDaYi77 commented 1 year ago

Hi QueuQ， I recently tried to reproduce the result you made，but i got a question. The last class acc was always 0% when i trained it. Can you help me to solve this problem？