Chapter06/02_dqn_pong.py can't learn properly after modified the ScaleFloatFrame

[liuyuezhang]

Hi, I am a reader of your book and recently met a very weird issue. I tried to modify your ScaledFloatFrame preprocessing, the only two modifications I made are:

• Firstly, in Chapter06/lib/wrappers.py, I commented the ScaledFloatFrame

  def make_env(env_name):
  env = gym.make(env_name)
  env = MaxAndSkipEnv(env)
  env = FireResetEnv(env)
  env = ProcessFrame84(env)
  env = ImageToPyTorch(env)
  env = BufferWrapper(env, 4)
  return env

• Secondly, in Chapter06/lib/dqn_model.py, I performed the scaling in the network model:

  def forward(self, x):
      fx = x.float() / 255.0
      conv_out = self.conv(fx).view(fx.size()[0], -1)
      return self.fc(conv_out)

  However, I found DQN can't learn properly on pong. Results are:

Any ideas of why such things happened? I just really don't know where I made the mistake...

[Shmuma]

If it was the only changes, it should work. Have you tried to rerun it several times? Sometimes (~10% or something) it fails to converge just because of an unlucky seed.