`ddqn.experience`'s `action` data type problem in `22_deep_reinforcement_learning`/`04_q_learning_for_trading.ipynb`

[DavidHJong]

Describe the bug

• When the train goes on for hundreds of episodes, it randomly generates a np.ndarray data type object in the ddqn.experience.action list. At very beginning, the interval is once per episode. Then, it becomes once per every 20 steps, and finally every step.

To Reproduce

1. Run the training for like 300 episodes.
2. Execute the below code, which will show a list of nth ddqn.experience whose action is not an integer.

   all_action_list = [ddqn.experience[i][1] for i in range(len(ddqn.experience))]
   for i in range(len(all_action_list)):
   if type(all_action_list[i]) != int:
       print(i)

Question

• Is this intended? Any purpose for this?

[stefan-jansen]

Based on your other issue, differences may be due to using more recent open AI gym versions, which generates what the current state (this_state = first entry in transition to memorize).

for episode in range(1, max_episodes + 1):
    this_state = trading_environment.reset()
    for episode_step in range(max_episode_steps):

Hope this helps.