Unable to reproduce results

[mattochal]

I am unable to reproduce results for the Consistent Networks paper. I think I may be misunderstanding something. Which algorithm should achieve the claimed 70.11% +- 0.19, and the 67.78% +- 0.20 in Table 3?

I am able to achieve the following averaged over 10 repeats on mini-ImageNet 5-shot 5-way: 66.79 +- 0.28 with basic-RW 66.94 +- 0.20 with basic-VAT-ENT, 64.64 +- 0.41 with kmeans-refine

The README file from the persistent_SSL branch claims basic-VAT-ENT should be the one to achieve the stated performance. Whereas the README file from the NIPS submission brach states I should be getting ~69.65% with basic-RW.

[AhmedAyad89]

What are the commands you are running for training and testing?

On Sat, 27 Jul 2019, 16:33 Mateusz Ochal, notifications@github.com wrote:

I am unable to reproduce results for the [ https://arxiv.org/abs/1903.02164](consistent networks paper). I think I may be misunderstanding something. Which algorithm should achieve the claimed 70.11% +- 0.19?

I am able to achieve the following averaged over 10 repeats: 66.79 +- 0.28 with basic-RW, 66.94 +- 0.20 with basic-VAT-ENT, 64.64 +- 0.41 with kmeans-refine

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AhmedAyad89/Consitent-Prototypical-Networks-Semi-Supervised-Few-Shot-Learning/issues/1?email_source=notifications&email_token=AC3TWO542DJOFJJPJVC7S5DQBRMB5A5CNFSM4IHJQYY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HB3HOUQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AC3TWO3BSWOZFQ5B2A47XT3QBRMB5ANCNFSM4IHJQYYQ .

[mattochal]

I'm saving the flags I am passing into the program (I have added some of my own flags as well but don't worry about those) { "data_root": "data", "load_images": true, "allstep": false, "VAT_epsilon": 8.0, "VAT_num_power_iterations": 1, "VAT_xi": 0.01, "graph_smoothing": 0.1, "visit_loss_weight": 1.0, "one_hop_weight": 0.7, "two_hop_weight": 0.49, "three_hop_weight": 0.35, "nhops": 10, "learn_radius": false, "init_radius": 100.0, "model": "basic-RW", "dataset": "mini-imagenet", "partition": null, "nshot": 5, "nclasses_eval": 5, "nclasses_train": 5, "num_test": 5, "results": "/media/disk1/mateusz/repositories/experiment_results/real-consistent-rw-and-kmeans-refine/consistent-protonet-basic-RW-mini-None-S00001", "gpu": "1", "max_steps": null, "steps_per_valid": null, "steps_per_log": null, "steps_per_save": null, "steps_per_summary": 200, "num_eval_episode": 200, "pretrain": null, "seed": 1, "classification_nshot": 5, "num_unlabel": 5, "num_distractor": 0, "label_ratio": 0.4, "overwrite": true, "save_with_model": false, "continue_train": false, "eval": false, "use_test": false, "learn_rate": null, "paper_setup": true, "paper_setup_small": false }

[AhmedAyad89]

For training: "VAT epsilon" should be set to 100, for the VAT or VAT-ENT models. "num_unlabel" should be set to 10 .

For testing: Make sure you set the "test" flag to true, this sa,ples from the test split rather than the validation spilt. If you want the 70% scenario, you test your trained VAT-ENT model, but with "model" set to kmeans-refine. This I think may be where the confusion is mainly coming from here. Set "num_unlabel" to 20, and make sure distractors are disabled, this should do it.

On Sat, Jul 27, 2019 at 7:08 PM Mateusz Ochal notifications@github.com wrote:

I'm saving the flags I am passing into the program (I have added some of my own flags as well but don't worry about those) { "data_root": "data", "load_images": true, "allstep": false, "VAT_epsilon": 8.0, "VAT_num_power_iterations": 1, "VAT_xi": 0.01, "graph_smoothing": 0.1, "visit_loss_weight": 1.0, "one_hop_weight": 0.7, "two_hop_weight": 0.49, "three_hop_weight": 0.35, "nhops": 10, "learn_radius": false, "init_radius": 100.0, "model": "basic-RW", "dataset": "mini-imagenet", "partition": null, "nshot": 5, "nclasses_eval": 5, "nclasses_train": 5, "num_test": 5, "results": "/media/disk1/mateusz/repositories/experiment_results/real-consistent-rw-and-kmeans-refine/consistent-protonet-basic-RW-mini-None-S00001", "gpu": "1", "max_steps": null, "steps_per_valid": null, "steps_per_log": null, "steps_per_save": null, "steps_per_summary": 200, "num_eval_episode": 200, "pretrain": null, "seed": 1, "classification_nshot": 5, "num_unlabel": 5, "num_distractor": 0, "label_ratio": 0.4, "overwrite": true, "save_with_model": false, "continue_train": false, "eval": false, "use_test": false, "learn_rate": null, "paper_setup": true, "paper_setup_small": false }

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AhmedAyad89/Consitent-Prototypical-Networks-Semi-Supervised-Few-Shot-Learning/issues/1?email_source=notifications&email_token=AC3TWOYQBAIOESBPI2EFHN3QBR6HNA5CNFSM4IHJQYY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26PFLI#issuecomment-515699373, or mute the thread https://github.com/notifications/unsubscribe-auth/AC3TWOZPZMP54I5ARED2MK3QBR6HNANCNFSM4IHJQYYQ .