PatrickHuembeli
commented
3 years ago Response to Reviewer #3
We would like to thank the reviewer for their in-depth comments. We implemented all their suggestions and we believe that thanks to this, the article gained substance and improved significantly.
Before answering all of the reviewer’s comments below, we summarize the main changes that we implemented:
- The title is changed and shortened.
- We clarified several aspects in the introduction and rewrote it entirely.
- We removed the notion of spins and parity throughout the article.
- We now refer to pattern storage or memorizing patterns instead of intelligent behaviour.
- We have revised Figure 2, illustrating the different architectures:
- We added information about the weights and nodes.
- The figure now shows the energy of the configurations.
- The figure now illustrates how hidden nodes can make a model more expressive.
- We shortened the final section substantially.
Answers to Reviewers comments
- “And Beyond” in the title is vague. Title is too long
We agree with the reviewer that “beyond” is vague and also not necessary in our title. We removed it and further shortened the title by removing “Equilibration and Machine Learning:”. The new title reads simply “The Physics of Energy-Based Models”.
- Is this a physics article or an ML article?
It is written as an interdisciplinary article with a target audience that includes members of both disciplines.
- Caption in hero figure has some grammatical errors
We thank the reviewer for this comment. We have fixed the error.
- “Using physics to understand the past and future of EBMs” makes this article sound like a historical work
We thank the reviewer for this suggestion. We have adapted this sentence to “Using physics to understand energy-based models”.
-
Introduction is grandiose. I like colorful introductions, but I’m worried that this one lacks substance. The authors speak vaguely of what they are going to do
- “Ultimately, we study modern concepts... venturing a glance into future research.”
- Makes this article sound speculative
- “This article should be understood as...”
- You shouldn’t have to tell the reader how the article is to be understood
- "...as energy functions, Boltzmann distributions, Ising models, Gibbs sampling, and contrastive divergence”
- They don’t connect the lines between these things, but rather toss them out. Not a huge fan of tossing out a bunch of concepts named after people, and which are thus unintuitive.
- Following up on this, after reading the remainder of the article:
- The article makes claims about sunrises, sailing, and intelligent agents. But all it does is denoise MNIST digits. Flashy introductions are great, but this one sets the bar too high, given the main contributions of the article.
- “Ultimately, we study modern concepts... venturing a glance into future research.”
We thank the reviewer for this valuable feedback. We have substantially rewritten and shortened the introduction following these suggestions.
- Energy-based models
- Spelling mistake: “uncontrolably”
We thank the reviewer for spotting this typo. We have fixed it in the revised version.
-
- “Harness randomness to create intelligent behavior” is not a good way to introduce EBMs. It’s unclear what this even means, or why there should be a connection between randomness and intelligent behavior. Also, denoising an image is hardly intelligent behavior.
We agree with the reviewer. We have adapted the phrasing throughout the article and now refer simply to “storing” or “memorizing” patterns.
- “Energy functions are also referred to as the Hamiltonian of the system.”
- This is an extreme simplification and very misleading. Using a Hamiltonian implies that you are using canonical coordinates as inputs to the energy function. This is not true in the vast majority of energy functions, especially in machine learning.
We thank the referee for pointing this out. While Hamiltonians can be defined beyond canonical coordinates, particularly in the context of quantum computing, we agree that our wording is potentially misleading and therefore have removed the mention of Hamiltonians in this section.
- A note on Boltzman distributions
- I’ve never thought that Boltzman distributions were special to physics. They partition a system into N cells and then normalize over those cells. They have a connection to physics in the same way that "parabolic arcs" (quadratic curves) do: they are so fundamental that, of course, they pop up in a wide array of physics problems. But you wouldn’t say that the quadratic formula is physics-inspired.
This is a very interesting point. We agree that the Boltzmann distribution is not special to physics. The distribution was first introduced in the context of statistical mechanics, but it indeed applies more widely. One of the primary aims of this article is that we can use physical systems to understand abstract concepts like probability distributions. Our strategy is akin to providing insights on quadratic curves by studying projectile motion, which can help deepen understanding.
We have updated the Boltzmann distribution section to clarify that the Boltzmann distribution was initially uncovered in statistical physics, but have made clear that it is more widely applicable across other fields.
- At this point, if I was a young reader, I would still have no idea what EBMs are
- Written at paragraph 11 or so
This is a very important point, and we thank the reviewer for providing this feedback. We have now added a definition for EBMs directly after the introduction.
- What audience is Figure 1 meant for?
- If it’s meant for ML readers, then “parity,” “spin” etc. need explaining. And I’m not sure that they are worth trying to explain here, as one can derive a Boltzman distribution without using quantum mechanics
- If it’s meant for physicists, the interactive demo is trivially true but the connection to ML and learning is absent.
We agree with the reviewer that the notion of parity required further explanation. Since it is not necessary for the understanding of this figure, we removed it entirely from the manuscript. We also added more explanation to the caption to make sure the figure can be understood independently from the article. Regarding “spin”, please see the answer below.
- A broader point about this article
- Why do we need to bring quantum mechanics into it at all? One can derive all of these algorithms without using quantum mechanics. If this article is trying to teach readers a new concept, it doesn’t make sense to use quantum mechanics (very unintuitive) when classical analogies (much more intuitive) will do just find.
In the original version of the article, we indeed use the word “spin” which may be potentially associated with quantum mechanics, even though there is also a classical notion of spin systems, like the Ising model. Nevertheless, we have removed the use of the word spin to avoid confusion.
- Ising model
- Ok, this is a natural progression from introducing RBMs
- “What intelligent tasks can be performed with these models?”
- They go on to describe denoising and memory recall.
- What we’ve done here is create attractor dynamics around several modes. And we initialize the network outside of the attractor and progressively move it towards a local min
- Not sure this is something we can construe as intelligent
- “Not bad for a group of spins” comes across as flippant.
We thank the reviewer for this feedback. We have updated the text following these suggestions.
- Figure 2
- Mediocre intuition for the three types of networks
- It was a good idea to put all 3 side-by-side. But the hidden unit color scheme is confusing. Also, the nodes and connections should be labeled with variables somehow, as we have spent a lot of effort introducing this notation in the text so far.
- There is trivial interactivity in that clicking on the nodes changes their spin. But I don’t think this helps much
We thank the reviewer for this suggestion, which has led us to improve the figure. First, we change the interactivity such that it becomes more clear that hidden nodes indeed allow more complex models. We use a simple symmetry breaking example for this. Furthermore, we changed the color of the unused hidden nodes for the Hopfield network. We have also added the names and values of the weights and nodes. This information is now shown when hovering over them with the mouse.
- Figure 3
- How was the plot constructed? Is that the true energy of the MNIST RBM they are sampling from (looks sinusoidal, which is more well-behaved than most energy landscapes at the MNIST level)
- Overall figure is nice. It doesn’t give me a ton of intuition, but I get the overall idea. The interactivity is useful here.
- The last sentence of the figure caption makes a strong claim, “This [ending up at non-local minima] can eventually allow the model to generalize beyond training data.” -> this is not a causal explanation. Maybe there is a long line of reasoning about this in some paper on RBMs, but this is unintuitive
We thank the reviewer for this feedback. We clarify in the caption that this is not the real energy landscape, but a simplified representation. We decided to remove the claim of generalization because the aim of the figure is to explain sampling from an energy landscape.
- Section subheadings would help a lot. I am starting to feel a bit lost as to where I am in the story and what the author is talking about now
- Aha, that section was about Gibbs sampling
- The discussion is reasonable, but wordy
We thank the reviewer for this suggestion. We have added a subheading for Gibbs sampling and added another one at the beginning of the article for the Boltzmann distribution.
- Typo “illustartes”
We corrected the typo, and thank the reviewer for catching it.
- I really enjoy the “Learning and Unlearning” figure
- This is the single best part of the paper
- It’s the sort of figure that Distill loves!
We thank the reviewer for their positive feedback on this figure.
- RBM learning demo that follows is quite nice. Although it overlaps a lot with previous figure
The idea to use the same figure as a teaser is to attract the reader, while at the end of the article the figure should be fully understood by the reader. So the figures are in fact the same, but we explain more details in the caption appearing in the main text. The caption of the teaser is very general and should not require any knowledge by the reader.
- No citations in first paragraph of “Future of” paragraph - some citations would be useful
Since we rewrote the whole final section, we decided to remove the “list” of possible research directions because we believe it does not add a lot of value. Therefore, we believe there are now no further citations required.
- Worth bringing up the curse of dimensionality in paragraph 3
We thank the reviewer for this suggestion. We have added a sentence about the curse of dimensionality.
- Type in quantum ML section: targetting
We corrected the typo. We thank the reviewer for spotting it.
- “hence severely limit” -> “hence severely limits”
We corrected the typo. We thank the reviewer for spotting it.
- They also talk about spin-glass systems and tensor networks Why are you talking about tensor networks without giving some intuition for what they are? Also, I’m not sure that any discussion of tensor networks belongs in this article, as they are sort of tangentially related to RBMs. This part was meant to give an outlook into ongoing research to give the advanced reader some of the keywords and references of where they could start getting involved with these advanced topics. Weagree with the reviewer that the discussion about tensor networks was not essential. We have rewritten and significantly shortened the final section.
Conclusions
- This post is long and opaque.
Following the suggestions of the reviewer, we have shortened the article and made several clarifications throughout. We believe the article is greatly improved as a consequence.
- The best part of it was the diagrams, some of which were quite good
We thank the reviewer for this feedback. We have also improved Figure 2 (architectures), which was lacking substance in the last version.
- The writing was grammatically correct and not wrong, but it was not particularly clear
Based on the reviewer’s comments, we have endeavoured to improve the clarity of the text. We have improved critical passages, especially in the introduction, which should make the article much clearer.
- The purpose of this sort of article is to communicate concepts at the intersection of Physics and ML really well. I worry that the way in which these concepts are introduced is overly complicated. For example, I see no advantage to discussing QM in this article because none of the algorithms described depend on quantum effects.
Following the reviewer's advice, we have shortened, clarified, and simplified the article. We note that now the only mention of quantum mechanics occurs in the last section, where we present quantum energy-models as a possible future research direction.
- Some of the visualizations need a better explanation. For example, the author is using synthetic curves to represent energy landscapes, rather than the true energy landscape of the model
We have added this to the Figures in question.
The following peer review was solicited as part of the Distill review process.
The reviewer chose to keep anonymity. Distill offers reviewers a choice between anonymous review and offering reviews under their name. Non-anonymous review allows reviewers to get credit for the service they offer to the community.
Distill is grateful to the reviewer for taking the time to review this article.
Specific notes
Conclusions
Distill employs a reviewer worksheet as a help for reviewers.
The first three parts of this worksheet ask reviewers to rate a submission along certain dimensions on a scale from 1 to 5. While the scale meaning is consistently "higher is better", please read the explanations for our expectations for each score—we do not expect even exceptionally good papers to receive a perfect score in every category, and expect most papers to be around a 3 in most categories.
Any concerns or conflicts of interest that you are aware of?: No known conflicts of interest
Any concerns or conflicts of interest that you are aware of?: No known conflicts of interest What type of contributions does this article make?: Explanation of existing results
Comments: This article explains some connections between physics and ML in a hard-to-follow manner: by bringing in quantum mechanics, making big promises in the introduction, and using more words than necessary to describe a given idea. It would be hard for a young researcher to learn, eg, EBMs starting from this article.