search

microsoft / Semi-supervised-learning

A Unified Semi-Supervised Learning Codebase (NeurIPS'22)
https://usb.readthedocs.io
MIT License
1.37k stars 182 forks source link

Cannot reproduce softmatch result on ag_news dataset #219

Closed Mezbaur-Rahman closed 3 months ago

Mezbaur-Rahman commented 6 months ago

I am running the following code:

python3 train.py --c config/usb_nlp/softmatch/softmatch_ag_news_40_0.yaml

But the accuracy I get is 0.8447368421 i.e error rate of 0.15 which is higher than what is reported in the result section (11.9±0.27). I am running the code with the exact config file from the repo. I also tried with different seeds but no improvement. What can be the issue here?

Hhhhhhao commented 6 months ago

Hi there.

Have a check at the log file here (from our last run): https://drive.google.com/drive/u/0/folders/1a9mWxDRiWBFbs0niAKNVAiUQyENFUghA

We merged several major pull request since then so it is possible that the parameters in the config file have been changed. You can check all the parameters we used in the log file, and also the final results.

Mezbaur-Rahman commented 6 months ago

I cannot find the use_amp parameter in my log file, it is also not present in the config file. Should I create a new argument named use_amp and set it to False? Also, the num_workers was set to 4 in my case.

Hhhhhhao commented 5 months ago

I cannot find the use_amp parameter in my log file, it is also not present in the config file. Should I create a new argument named use_amp and set it to False? Also, the num_workers was set to 4 in my case.

I believe that argument has been changed in the past commits. Now we might be using --amp, can you have a check on that? But I think both 'amp' and 'num_workers' wouldn't affect the performance too much.

github-actions[bot] commented 3 months ago

Stale issue message