Open jeromepl opened 5 years ago
Hi, please find below a review submitted by one of the reviewers:
Score: 5 Reviewer 3 comment : PROBLEM STATEMENT The problem is very clearly presented in a self-contained fashion.
CODE The whole code is presented in a Google Colab notebook. This allows an easy understanding of the code. On the other hand, the notebook does not contain any graph as the ones presented in the report. Maybe this is related to the limited computation resources available for the free Colab notebooks.
COMMUNICATION WITH THE ORIGINAL AUTHORS There is no reference to communication with authors.
HYPERPARAMETER SEARCH There is no explicit mention to the work on trying to obtain better hyperparameters.
They do though introduce a study of the impact of adding biases in the normalization as well.
ABLATION STUDIES No ablation studies were reported.
DISCUSSION ON RESULTS The obtained results are basically commented. Results match the ICLR submission but for the deepest network, but as the same authors state, this could be related to the variance involved when running a single training of the model due to low computational power.
However, I do not understand why Batch Norm is not included in Figure 3.
RECOMMENDATIONS FOR REPRODUCIBILITY No comments in this sense.
OVERALL ORGANIZATION AND CLARITY The paper is clear to follow and understand. The authors made an effort to make it self-contained.
My main criticism is on Figure 1, which is actually taken from the ICLR submission. Firstly, citation should be added and, secondly, it is almost impossible to compare the relevant curves of Figure 1 with the ones obtained in Figure 2. Only those plots being reproduced should be included in a single Figure comparing the reproducibility and the original values. This central result is totally missing in the report, so I encourage the authors to create a graph with a fair comparison of the obtained results and the ones reported in the ICLR submission.
Confidence : 4
Hi, please find below a review submitted by one of the reviewers:
Score: 5 Reviewer 1 comment : Problem Statement
Equi-Normalization is well-justified in the Introduction of the reproduced paper.
Code
The code is open-source. All the code is in one Jupyter notebook. Cannot comment much about the codebase itself, but the structure and the comments looks good.
Communication with the Original Author
The authors mention that they have communicated with the authors via OpenReview. The communication can be found in OpenReview.
Hyperparameter Search
No hyperparameter search was performed.
Ablation Study
Ablation study of adding bias to the neural network was performed. No other ablation study is given, other than the ones already in the original paper (BN, ENorm, BN+ENorm).
Discussion on Results
The authors give an adequate comparison between published and reproduced results. There could be more speculations about the the p=19 case.
Recommendations for Reproducibility
There was no suggestions to the author about possible improvements in terms of reproducibility.
Overall Organization and Clarity
Overall, the paper was written coherently.
Here are points that I was particularly impressed with:
Here are some parts of the paper I wished for more information:
Here are some minor fixes I recommend:
Thank you!
Score (1~10): 5 Confidence Score (1~5): 3 Confidence : 3
Hi, please find below a review submitted by one of the reviewers:
Score: 4 Reviewer 2 comment : Code is nicely commented and easy enough to read. Confidence : 3
Reviewer 2 comment : In detail, it would have been nice to take advantage of notebook format as to show the reproduced results. In present state, it is a bit hard to judge as to what the output would look like.
Submission for #95