Martyn0324
commented
2 years ago It seems that this error will only happen when using the arguments for RainbowDQNAgent.current_episode. agent.current_step runs just fine.
After those changes are made and the model is saved, it won't work again, so it's necessary to delete the saved model weights and start the training all over again.
I checked the RainbowDQNAgent file:
def generate_actions(self, state, **kwargs):
frames = list()
for game_frame in state.frames:
frames.append(torch.tensor(torch.from_numpy(game_frame.frame), dtype=torch.float32, device=self.device))
self.current_state = torch.stack(frames, 0)
if self.mode == RainbowDQNAgentModes.OBSERVE and self.observe_mode == "RANDOM":
self.current_action = random.randint(0, len(self.game_inputs[0]["inputs"]) - 1)
elif self.mode == RainbowDQNAgentModes.OBSERVE and self.observe_mode == "MODEL":
self.agent.reset_noise()
self.current_action = self.agent.act(self.current_state)
elif self.mode == RainbowDQNAgentModes.TRAIN:
self.agent.reset_noise()
self.current_action = self.agent.act(self.current_state)
actions = list()
label = self.game_inputs_mappings[0][self.current_action]
action = self.game_inputs[0]["inputs"][label]
value = self.game_inputs[0]["value"]
actions.append((label, action, value))
for action in actions:
self.analytics_client.track(
event_key="AGENT_ACTION",
data={
"label": action[0],
"action": [str(a) for a in action[1]],
"input_value": action[2]
}
)
return actionsAt first, I thought there would be any relation between self.current_state and self.current_episode. But there is not.
I have no idea what the variable agent.current_episode has to do with self.current_action = self.agent.act(self.current_state).
It seems that any kind of change made to the variable self.current_episode causes this problem.
However, from all I can see in /serpent/machine_learning/reinforcement_learning/agents/rainbow_dqn_agent, this variable is only responsible for the JSON file AND when using the function observe with terminal = True AND self.mode == RainbowDQNAgentModes.TRAIN. I don't use the TRAIN mode, though(It's basically the same thing as using the OBSERVE mode)
EDIT: In fact, agent.current_episode has nothing to do with self.current_action. I don't remember why I did this association.
If you're having problems with this variable, you might be looking for this issue
Expected result
Continue training the RainbowDQN Agent, resuming training for the 4th time. This time, registering in the JSON file the correct run number and its steps.
Encountered result
Steps to reproduce
Change the code, adding to the
setup_playfunction the following lines:Note: While this problem happens when loading a model weights for RainbowDQN, this happens ALL THE TIME when using PPO Agent. Note²: This also seems to happen when you change your game inputs. This isn't the case right now. *Note³: This issue doesn't seem to affect the AI performance at all. It's purely an "aesthetic" issue.