Submission for issue #77

[shishir13sharma]

77

[reproducibility-org]

Hi, please find below a review submitted by one of the reviewers:

Score: 7 Reviewer 1 comment : This work tries to reproduce Q-Map on several benchmark games. Although the results of reproduction are negative, the authors offer analysis and detailed results. The report is well-written, but the code documentation is terrible (for example, markdown format of README is incorrect). More hyper-parameter search can be conducted. Even though, I would give a weak accept for encouragement of participating reproduction challenge. Confidence : 3

[reproducibility-org]

Hi, please find below a review submitted by one of the reviewers:

Score: 5 Reviewer 3 comment : This report describes an attempt to reproduce the results in the paper "Q-map: a Convolutional Approach for Goal-Oriented Reinforcement Learning".

• The report is clearly written. The authors described the paper and the experiments they run in detail.

• The authors seem to have written their own code for the paper (in pytorch) and I appreciate the effort.

• While I understand the resource limitations on running these experiments. The experiments still seem to be fairly limited. The experiments would have been more information if it multiple runs over different hyper parameters were performed.

• The authors found negative results. I am curious if the authors have attempted to reach out to the original authors to clarify details (there could have been slight differences in implementation that effects the performance of the method drastically).

Overall, i appreciate the effort. However, I would like to see more extensive experiments in order to make a better judgement on the reproducibility of the original paper. Confidence : 4