Reinforcement_Learning.py - log_q_values.read()

[f-26]

Hi Magnus, first of all, I wanted to thank you for providing such a huge amount of knowledge for free. I am currently working on a research paper for using reinforcement learning in a strategic planning simulation and your tutorials helped me quite a lot already. Even though I am trying to do most of the work myself I am using your work as a reference and I have learned much more than I would have ever expected from a free resource like this.

Atm I am trying to run your model to compare its performance to my work but I ran into a problem. When I execute the line: log_q_values.read() log_reward.read()

in google colab I get the following error:

FileNotFoundError Traceback (most recent call last)

in () ----> 1 log_q_values.read() 2 log_reward.read() /content/reinforcement_learning.py in read(self) 367 368 # Read the log-file using the super-class. --> 369 self._read() 370 371 # Get the logged statistics for the Q-values. /content/reinforcement_learning.py in _read(self) 267 268 # Open and read the log-file. --> 269 with open(self.file_path) as f: 270 reader = csv.reader(f, delimiter="\t") 271 self.count_episodes, self.count_states, *data = zip(*reader) FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints_tutorial16/Breakout-v0/log_q_values.txt' However, this is not the only problem. Your model seems to be stuck at training without rly improving. Q-values are oscillatng around .02 without rly chaning much. I would really appreciate any help. Kind regards, Felix

[Hvass-Labs]

I'm glad that you found my work useful! You are quite right that it was a huge effort to make all this!

I am not sure if you are running a modified version - if that is the case, then unfortunately I cannot help you, as I cannot give individual support on mods (look at the user-count - I would spend my whole life on doing that).

If you are running the code straight from my github repo without any mods, then I would suggest running it on your local machine, because Colab might have some limitations on file access etc.

You can also try and disable the log-writing. I think it was about 2 years since I made this, so I obviously can't remember the details. But I would imagine that you can just disable the logging, so it won't read/write to files.

I don't know why the training progress is stuck. As I recall, it improves fairly rapidly within the first few hours of running, and then the progress slows down.

I am closing this issue because I don't think I can help you. But if you find a solution to your problem, then please make a note here, so others might be helped in the future.

Good luck!

[Hvass-Labs]

I also encountered this problem on a fresh install of these tutorials on a clean machine.

The reason is that by default the Notebook only trains for 1 episode and does not create a log-file. The intention is that you run the actual training in a terminal with the following command because it can take many hours or even days:

python reinforcement_learning.py --env Breakout-v0 --training

You can try running the training entirely in the Notebook (e.g. on Colab). You need to modify a few things. First you need to find these lines and change it to use_logging=True:

agent = rl.Agent(env_name=env_name,
                 training=True,
                 render=True,
                 use_logging=False)

Then you need to increase the number of episodes in the following line. Note that this line occurs multiple times in the Notebook and you should change the line in the Training-section:

agent.run(num_episodes=1)

You can try 1000 episodes to start with. I'm not sure how many episodes I ran during training, but it appears that it might have been close to 100k, which is why it is best to run it in a terminal.

I have tried running this both locally on my Linux PC and on Google Colab. It seems to be working both places, but Colab is VERY slow.