use of target_network in duel.py (in example)

[pavitrakumar78]

Hi,

I have been exploring your example code of dual networks in dqn. In that, I don't see how the target network is really used in your implementation. Its weights are updated at constant intervals, but you don't "use" it anywhere. The existence of target network is to compute the target_q values right? which is 'qpost' in your code if I am not mistaken. So, shouldn't line 144 be

qpost = target_model.predict(poststates)

instead of

qpost = model.predict(poststates)

Please correct me if I'm wrong, I am still learning about dqns! :)

[tambetm]

Oh crap, you are right! I don't know how this bug slipped through. I remember spending elaborate amounts of time tuning tau and that was all for nothing? Can't believe it...

I don't have access to Minecraft on computer I'm on, but I tested with my original CartPole code and it seems to make learning slower, but more stable. Basically you can achieve previous result by setting --tau=1.

Let me know if the duel code is working for you. I tested it in very initial stages and don't really remember the results.

[pavitrakumar78]

I also didn't notice it when I was testing on Minecraft envs..! I will let you know how it performs. The code (with the bug) was working (learning happens) for Minecraft basic, I've yet to test it for other minecraft envs.

[pavitrakumar78]

@tambetm Did you try the duel.py code on any env other than MinecraftBasic? I am having a hard time training for any other environment - Agent trying to learn CliffWalking-Level doesn't seem to show any learning. I am trying it on Eating1-Level at the moment.

Also, another issue is that is it possible to view the testing in a higher resolution? If we do the training in 40x30 size, is there a way to view the testing (as a video) in a higher resolution? I tried resizing higher resolution to 40x30 size, but that gives bad testing performance.

[tambetm]

I'm not sure I actually got duel.py doing anything reasonable. Maybe I should delete it from repo...

I'm using following wrapper in my code:

from gym import ObservationWrapper
import cv2

class ResizeWrapper(ObservationWrapper):
    def __init__(self, env, width, height):
        assert isinstance(env.observation_space, Box)
        super(ResizeWrapper, self).__init__(env)
        self.width = width
        self.height = height
        self.observation_space = Box(0, 255, (height, width) + env.observation_space.low.shape[2:])

    def _observation(self, observation):
        return cv2.resize(observation, (self.width, self.height))

With this wrapper you can render Minecraft in full resolution (say 640x480) and env.render() method shows it like that. But agent observation is resized to 40x30 (or whatever arguments you gave to wrapper constructor).

[pavitrakumar78]

Ah, I used a similar method to get it working! :) Your method is a nice and neat way of doing it! Thank you for the suggestion!

[tambetm]

Deleted duel.py and buffer.py from the project, because they were not really working.