titu1994
commented
5 years ago So I'll preface this with two things, one that this would be my first foray into RL and that the paper was woefully inadequate in implementation details.
1) My understanding from the paper is that during evaluation, they use argmax to select the actual action (in this case cell type or filter count etc). I considered using the TF.multinomial to sample during training and then test with argmax, but eventually gave up on the idea as it would add significant complexity. In hindsight, it makes sense to do this, but I got somewhat stable results without it.
2) Didn't know this was an issue. In hindsight, this is what I do (I think) in my Progressive NAS implementation, so it should be done here as well. I would appreciate a PR.
3) This is from the paper, the state is chained in for each classifier to make it semantically dependent on the previous prediction. (at least that's the idea)
4) Discounting was just something I found to stabilise training a lot. If I remember the theory, the mathematical requirement is to make the infinite series converge eventually. In the case of code implementation, I just found it to help training significantly. NAS was highly unstable without it. So that's my empirical explanation. I would like if someone more knowledgeable will explain in detail why and how it helps here
Yes I modeled the entire thing as episodic memory style training, cause that's what I was familiar with (my knowledge is mainly from Deep Minds papers on AlphaGo, Atari DQNs etc so I transfered what I knew to this).
Of course, the original authors might have used something more advanced (I am pretty sure they did), and therefore their implementation would be vastly superior to mine.
brendenpetersen
Hi, thank you for releasing this implementation! I have several questions regarding how this works relative to my understanding of the original paper:
Why is the action always the mode (i.e. argmax of the logits) of the policy each iteration, rather than sample from the policy? The idea behind policy gradients it that you sample an action from a policy. Is this because batch size is 1, so rather than sampling you just take the mode?
You also take a random/exploratory action with probability epsilon. However, the logprob (and thus loss) of exploratory actions seem to be computed incorrectly. Your loss uses
self.policy_classifiersto compute the logprob; however, these tensors are conditioned on the argmax of the policy, not the action that was actually taken. The loss should always use the logprob of the action taken (in this case the exploratory one), not the argmax of the policy.Each time you generate a list of actions (i.e. a single architecture), you start by feeding in the state from the previous architecture (L78 of train.py). Is this correct, and if so what is the reason for this? Each iteration the controller has new parameters and will thus emit different architectures. Feeding the controller the previous state as
I don't understand the discounting, or rather, why discounting is performed at all. A single architecture gives you a single reward, in a sort of "1-step" episode process.
Based on 3 and 4, it seems like you view the entire process as a single RL "episode," as opposed to each architecture comprising its own episode. Is this correct?
I understand the original paper is very unclear on a lot of these points, so I appreciate any feedback or discussion!