search

diningphil / gnn-comparison

Official Repository of "A Fair Comparison of Graph Neural Networks for Graph Classification", ICLR 2020
GNU General Public License v3.0
385 stars 54 forks source link

The position of "config_id += 1" in the 84 of evaluation/model_selection/HoldOutSelector.py #9

Closed JennyyiqiW closed 4 years ago

JennyyiqiW commented 4 years ago

Would it be better if we put "config_id += 1" in line 72, instead of 84? In the case where the process is killed when only several folds results are finished, and we want to rerun the process without overwriting the previous folds results, "continue" in the line 82 would make the process skip "config_id += 1", resulting in the error below:

File cuda_results/diffpool/ENZYMES/baseline/DiffPoolssl_ENZYMES_assessment/10_NESTED_CV/OUTER_FOLD_1/outer_results.json already present! Shutting down to prevent loss of previous experiments Config cuda_results/diffpool/ENZYMES/baseline/DiffPoolssl_ENZYMES_assessment/10_NESTED_CV/OUTER_FOLD_2/HOLDOUT_MS/config_1/config_results.json already present! Shutting down to prevent loss of previous experiments Traceback (most recent call last): File "Launch_Experiments.py", line 41, in raise e # print(e) File "Launch_Experiments.py", line 38, in result_folder=args.result_folder,label_ratio=args.label_ratio,ssl_option=args.ssl_option, debug=args.debug) File "/mnt/ufs18/home-118/wangy206/torch-projects/self-surpervised-GNN/ssl-verification-1/EndToEnd_Evaluation.py", line 30, in main risk_assesser.risk_assessment(experiment_class, debug=debug) File "/mnt/ufs18/home-118/wangy206/torch-projects/self-surpervised-GNN/ssl-verification-1/evaluation/risk_assessment/K_Fold_Assessment.py", line 92, in risk_assessment self._risk_assessment_helper(outer_k, experiment_class, kfold_folder, debug, other) File "/mnt/ufs18/home-118/wangy206/torch-projects/self-surpervised-GNN/ssl-verification-1/evaluation/risk_assessment/K_Fold_Assessment.py", line 119, in _risk_assessment_helper self.model_configs, debug, other) File "/mnt/ufs18/home-118/wangy206/torch-projects/self-surpervised-GNN/ssl-verification-1/evaluation/model_selection/HoldOutSelector.py", line 102, in model_selection best_config = self.process_results(HOLDOUT_MS_FOLDER, config_id) File "/mnt/ufs18/home-118/wangy206/torch-projects/self-surpervised-GNN/ssl-verification-1/evaluation/model_selection/HoldOutSelector.py", line 46, in process_results print('Model selection winner for experiment', HOLDOUT_MS_FOLDER, 'is config ', best_i, ':') UnboundLocalError: local variable 'best_i' referenced before assignment

diningphil commented 4 years ago

Perhaps you are right. Still, this repository is exclusively for reproducibility purposes, so we would like to keep things as similar as they were at the time of the submission.

Thanks for raising this point though, it may help other people in improving the library for their own purposes.