Editorial decision for Distill | 2020 | Distill Reviewers
Reviewer 1 (mostly 4s, a few 3s and 5s)
Several technical comments, well-motivated, which the authors responded to, in some cases making significant fixes to the article. Closing line of review:
“Overall, I think this paper provides a valuable method and example of understanding visual features learned in an RL model and their interpretability. It’s a valuable contribution to the RL community.”
Reviewer 2 (mostly 4s, a few 3s and 5s)
Overall very positive. Focused on the diversity hypothesis and wording around several of the paper’s main claims (eg, suspicious of “if” direction on diversity hypothesis "diverse distributions <-> interpretable models"). I agree with this comment. Authors addressed it properly with changes to the text. Authors responded to other comments as well.
“Overall, this article presents important contributions, and communicates them clearly and correctly.”
Reviewer 3 (mostly 4s, a few 5s)
Concerns about “...forward-referencing the technique used for ""model editing"" -- makes it hard to peer review. Brings this up twice. This is an important criticism. I double-checked the current draft of the article, and these references appear to be removed. Good. Also, I found the description of weight editing in the Appendix to be satisfactory.
“It is very nice / intriguing how effective dimensionality reduction is here.”
Editorial decision
This was an informative article with clear, well-designed figures and good supporting text. It tackles a significant problem -- understanding RL vision -- and advances the state of the art quite a bit. We appreciated the multiple angles of attack which vary from attribution to feature visualization to spatially-aware feature visualization. The model editing results did an excellent job of confirming the validity of the feature visualization techniques. One key weakness is that it does not specifically have a “Related Work” section which would have put this work in context better. The authors ameliorate this problem by referencing related work extensively throughout the rest of the article. Another weakness is that the “Questions for further research” section is long, overly cautious and vague, and does not send a strong takeaway to the reader. Fortunately, this section occurs at the end of the article, so it will not deter readers from reading the really important bits. Apart from these relatively minor criticisms, this article is in good shape. It makes significant scientific contributions and is overall clear and well-written. Our decision is: accept.
Other
Important nit: prevent videos from auto-playing - this is distracting to the reader.
Understanding RL Vision
Editorial decision for Distill | 2020 | Distill Reviewers
Reviewer 1 (mostly 4s, a few 3s and 5s) Several technical comments, well-motivated, which the authors responded to, in some cases making significant fixes to the article. Closing line of review: “Overall, I think this paper provides a valuable method and example of understanding visual features learned in an RL model and their interpretability. It’s a valuable contribution to the RL community.”
Reviewer 2 (mostly 4s, a few 3s and 5s) Overall very positive. Focused on the diversity hypothesis and wording around several of the paper’s main claims (eg, suspicious of “if” direction on diversity hypothesis "diverse distributions <-> interpretable models"). I agree with this comment. Authors addressed it properly with changes to the text. Authors responded to other comments as well. “Overall, this article presents important contributions, and communicates them clearly and correctly.”
Reviewer 3 (mostly 4s, a few 5s) Concerns about “...forward-referencing the technique used for ""model editing"" -- makes it hard to peer review. Brings this up twice. This is an important criticism. I double-checked the current draft of the article, and these references appear to be removed. Good. Also, I found the description of weight editing in the Appendix to be satisfactory. “It is very nice / intriguing how effective dimensionality reduction is here.”
Editorial decision This was an informative article with clear, well-designed figures and good supporting text. It tackles a significant problem -- understanding RL vision -- and advances the state of the art quite a bit. We appreciated the multiple angles of attack which vary from attribution to feature visualization to spatially-aware feature visualization. The model editing results did an excellent job of confirming the validity of the feature visualization techniques. One key weakness is that it does not specifically have a “Related Work” section which would have put this work in context better. The authors ameliorate this problem by referencing related work extensively throughout the rest of the article. Another weakness is that the “Questions for further research” section is long, overly cautious and vague, and does not send a strong takeaway to the reader. Fortunately, this section occurs at the end of the article, so it will not deter readers from reading the really important bits. Apart from these relatively minor criticisms, this article is in good shape. It makes significant scientific contributions and is overall clear and well-written. Our decision is: accept.
Other Important nit: prevent videos from auto-playing - this is distracting to the reader.