Questions about results figures

[narutoten520]

@eleurent Hi eleurent, sorry to bother you again. I am not sure you receive my yesterday question or not. So I want to try again. With your help, now I can train and test the Highway-env with DQN. However, when I follow your instruction to see the results with tensorboard, it shows the following error:

(base) C:\Users\user1\rl-agents\scripts>tensorboard --logdir out/HighwayEnv/DQNAgent/run_20200619-114451_13448 TensorBoard 1.15.0 at http://DESKTOP-H6H55DR:6006/ (Press CTRL+C to quit) W0619 11:45:42.685292 19736 core_plugin.py:185] Unable to get first event timestamp for run .: No event timestamp could be found.

I am not sure my training process is right or not because it finishes with only sevaral seconds with 5000 episodes (I am wondering there may no results stored in run-files ). So I attach the run-files to let you check. Thanks for your help and look forward to your response. Best!

run_20200619-114221_19036.zip

[narutoten520]

I am wondering you may want to see the training process is right or not. So I attach them as follow. Please help check. Many thanks.

(base) C:\Users\user1\rl-agents\scripts>python experiments.py evaluate configs/HighwayEnv/env.json configs/HighwayEnv/agents/DQNAgent/dqn.json --train --episodes=5000 pygame 1.9.6 Hello from the pygame community. https://www.pygame.org/contribute.html INFO: Making new env: highway-v0 [INFO] NumExpr defaulting to 8 threads. [WARNING] Preferred device cuda:best unavailable, switching to default cpu INFO: Creating monitor directory out\HighwayEnv\DQNAgent\run_20200620-121952_24396 INFO: Starting new video recorder writing to C:\Users\user1\rl-agents\scripts\out\HighwayEnv\DQNAgent\run_20200620-121952_24396\openaigym.video.0.24396.video000000.mp4 Traceback (most recent call last): File "experiments.py", line 148, in main() File "experiments.py", line 43, in main evaluate(opts[''], opts[''], opts) File "experiments.py", line 77, in evaluate evaluation.train() File "D:\Anaconda\lib\site-packages\rl_agents\trainer\evaluation.py", line 106, in train self.run_episodes() File "D:\Anaconda\lib\site-packages\rl_agents\trainer\evaluation.py", line 131, in run_episodes reward, terminal = self.step() File "D:\Anaconda\lib\site-packages\rl_agents\trainer\evaluation.py", line 162, in step self.observation, reward, terminal, info = self.monitor.step(action) File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\gym\wrappers\monitor.py", line 31, in step observation, reward, done, info = self.env.step(action) File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\highway_env.py", line 48, in step return super().step(action) File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 179, in step self._simulate(action) File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 219, in _simulate self._automatic_rendering() File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 298, in _automatic_rendering self.automatic_rendering_callback() File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\gym\wrappers\monitoring\video_recorder.py", line 116, in capture_frame self._encode_image_frame(frame) File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\gym\wrappers\monitoring\video_recorder.py", line 162, in _encode_image_frame self.encoder = ImageEncoder(self.path, frame.shape, self.frames_per_sec, self.output_frames_per_sec) File "C:\Users\user1\AppData\Roaming\Python\Python37\site-packages\gym\wrappers\monitoring\video_recorder.py", line 255, in init raise error.DependencyNotInstalled("""Found neither the ffmpeg nor avconv executables. On OS X, you can install ffmpeg via brew install ffmpeg. On most Ubuntu variants, sudo apt-get install ffmpeg should do it. On Ubuntu 14.04, however, you'll need to install avconv with sudo apt-get install libav-tools.""") gym.error.DependencyNotInstalled: Found neither the ffmpeg nor avconv executables. On OS X, you can install ffmpeg via brew install ffmpeg. On most Ubuntu variants, sudo apt-get install ffmpeg should do it. On Ubuntu 14.04, however, you'll need to install avconv with sudo apt-get install libav-tools. INFO: Finished writing results. You can upload them to the scoreboard via gym.upload('C:\Users\user1\rl-agents\scripts\out\HighwayEnv\DQNAgent\run_20200620-121952_24396')

[eleurent]

Hi @narutoten520 It seems the training process is interrupted because it is trying to record videos, but you are missing the ffmpeg / avconv video utilities, so an error is raised.

You can install them with sudo apt-get install ffmpeg or sudo apt-get install libav-tools, as mentionned in the error message. EDIT: since you're on Windows, you should install ffmpeg.exe manually and add it the the PATH environment variable (so it can be found by python)

Alternatively, you can run the training with the --no-display option, which disables rendering altogether.

[narutoten520]

Hello @eleurent , many thanks for your guidance, now I can run and test the toolbox now. I checked other questions and found no answers, so I have to bother you again, sorry about that. I have some questions about how to store the training results, hope you can help me:

1. I can use tensorboard to see the reward, can I store them by adding some codes somewhere, so I can plot my own style of these figures.

2. In the highway-env, can I store the speed, acceleration and position of the ego vehicle in each episode? I also want plot these figures to see the training performance.

3. I also want to store the control actions of ego vehicle in each episodes, can I add some codes somewhere to realize that?

4. Moreover, can I know the position and speed of the surrounding vehicles, I want to see when the ego vehicle overtake them.

Thanks for your help in advance. Best regards.

[eleurent]

Note for potential readers: these questions are also related to the rl-agents project.

Hi @narutoten520,

1. For every run, the environment is wrapped in a gym Monitor, which means the main statistics of each episode (rewards, lengths, seed) are logged to an episode_batch.*.stats.json file in the run directory. Some example code for loading and plotting the data is already available in the scripts/analyze.py script, so you can work from there.
2. The info dict returned by env.step(action) contains some information that are logged in the gym Monitor stats file. The speed is currently available, but you would need to modify the returned info to add acceleration and position.
3. The control actions are not logged by default, but again you could add them to the info dict returned by the environemnt in the env.step(action) method.
4. The position and speed of surrounding vehicles can be accessed through [vehicle.position, vehicle.speed for vehicle in env.road.vehicles], but you would need to log them yourself.

Does that help?

[narutoten520]

Hello @eleurent , your answers are very helpful for me without a doubt. I need to understand them one by one slowly as I am a beginner. Today, I was tucked by the analyze.py. I have tried many ways to run it with your usage explanation, but I failed. Please help me. The commands I have tried:

1. python analyze.py run out/HighwayEnv/DQNAgent/run_20200625-221556_12864... --out Figures
2. python analyze.py run out/HighwayEnv/DQNAgent/run_20200625-221556_12864/openaigym.episode_batch.0.12864.stats.json... --out
3. python analyze.py run out/HighwayEnv/DQNAgent/run_20200625-221556_12864/openaigym.episode_batch.0.12864.stats.json... --out=Figures ("Figures is a folder")
4. python analyze.py run out/HighwayEnv/DQNAgent/run_20200625-221556_12864/openaigym.episode_batch.0.12864.stats.json... --last=10
5. python analyze.py run out/HighwayEnv/DQNAgent/run_20200625-221556_12864/openaigym.episode_batch.0.12864.stats.json... --out Figures

Many error information is the same as: (base) C:\Users\tengl\rl-agents\scripts>python analyze.py run out/HighwayEnv/DQNAgent/run_20200625-221556_12864/openaigym.episode_batch.0.12864.stats.json... --out Figures INFO:root:Fetching data in out\HighwayEnv\DQNAgent\run_20200625-221556_12864\openaigym.episode_batch.0.12864.stats.json... INFO:root:Found 0 runs. Traceback (most recent call last): File "D:\Anacoda\lib\site-packages\pandas\core\indexes\base.py", line 2646, in get_loc return self._engine.get_loc(key) File "pandas_libs\index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas_libs\hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'episode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "analyze.py", line 134, in main() File "analyze.py", line 36, in main RunAnalyzer(opts[''], out=opts["--out"], episodes_range=episodes_range) File "analyze.py", line 49, in init self.analyze() File "analyze.py", line 100, in analyze self.find_best_run() File "analyze.py", line 114, in find_best_run df = df[df["episode"] == df["episode"].max()].sort_values(criteria, ascending=ascending) File "D:\Anacoda\lib\site-packages\pandas\core\frame.py", line 2800, in getitem indexer = self.columns.get_loc(key) File "D:\Anacoda\lib\site-packages\pandas\core\indexes\base.py", line 2648, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas_libs\index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas_libs\hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'episode'

Please help me to find the right command to use the analyze.py file. Thanks.

[eleurent]

This script is actually very misleading, sorry about that. I'm not really using it anymore (just tensorboard) and was in fact thinking of removing it. The answer is that you should actually give the parent directory that contains your runs, so that they can be averaged. (Yes I know, it's not what is described)

So simply try: python analyze.py run out/HighwayEnv/DQNAgent

But this script should be fixed / better documented / removed at some point.

[narutoten520]

Hello @eleurent , Many thanks for your response and sorry for troubling you again. When I use "configs/HighwayEnv/agents/DQNAgent/ego_attention.json" this agent to train any Highway-env, there will be an issue. Hope you can help me!

1. When I use "ego_attention.json" to train "configs/HighwayEnv/env_obs_attention.json" this environment, the problem is:

(base) C:\Users\tengl\rl-agents\scripts>python experiments.py evaluate configs/HighwayEnv/env_obs_attention.json configs/HighwayEnv/agents/DQNAgent/ego_attention.json --train --episodes=10 INFO: Making new env: highway-v0 [INFO] Choosing GPU device: 0, memory used: 195 INFO: Creating monitor directory out\HighwayEnv\DQNAgent\run_20200629-114414_14436 INFO: Starting new video recorder writing to C:\Users\tengl\rl-agents\scripts\out\HighwayEnv\DQNAgent\run_20200629-114414_14436\openaigym.video.0.14436.video000000.mp4 Traceback (most recent call last): File "experiments.py", line 152, in main() File "experiments.py", line 46, in main evaluate(opts[''], opts[''], opts) File "experiments.py", line 81, in evaluate evaluation.train() File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\trainer\evaluation.py", line 107, in train self.run_episodes() File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\trainer\evaluation.py", line 135, in run_episodes reward, terminal = self.step() File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\trainer\evaluation.py", line 166, in step self.observation, reward, terminal, info = self.monitor.step(action) File "D:\Anacoda\lib\site-packages\gym\wrappers\monitor.py", line 31, in step observation, reward, done, info = self.env.step(action) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\highway_env.py", line 54, in step return super().step(action) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 165, in step self._simulate(action) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 198, in _simulate self._automatic_rendering() File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 281, in _automatic_rendering self.automatic_rendering_callback() File "D:\Anacoda\lib\site-packages\gym\wrappers\monitoring\video_recorder.py", line 101, in capture_frame frame = self.env.render(mode=render_mode) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\abstract.py", line 221, in render self.viewer.display() File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\highway_env\envs\common\graphics.py", line 112, in display self.agent_display(self.agent_surface, self.sim_surface) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\trainer\evaluation.py", line 93, in lambda agent_surface, sim_surface: AgentGraphics.display(self.agent, agent_surface, sim_surface)) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\common\graphics.py", line 32, in display DQNGraphics.display(agent, agent_surface, sim_surface) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\deep_q_network\graphics.py", line 50, in display cls.display_vehicles_attention(agent, sim_surface) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\deep_q_network\graphics.py", line 58, in display_vehicles_attention cls.v_attention = cls.compute_vehicles_attention(agent, state) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\deep_q_network\graphics.py", line 97, in compute_vehicles_attention v_feature = state[v_index, agent.env.observation.features.index(feature)] AttributeError: 'HighwayEnv' object has no attribute 'observation' INFO: Finished writing results. You can upload them to the scoreboard via gym.upload('C:\Users\tengl\rl-agents\scripts\out\HighwayEnv\DQNAgent\run_20200629-114414_14436')

2. When I use "ego_attention.json" to train other environments in the Highway-env, the error is the same as follows:

(base) C:\Users\tengl\rl-agents\scripts>python experiments.py evaluate configs/HighwayEnv/env.json configs/HighwayEnv/agents/DQNAgent/ego_attention.json --train --episodes=10 INFO: Making new env: highway-v0 [INFO] Choosing GPU device: 0, memory used: 195 INFO: Creating monitor directory out\HighwayEnv\DQNAgent\run_20200629-114727_13208 size mismatch, m1: [1 x 5], m2: [7 x 64] at C:/cb/pytorch_1000000000000/work/aten/src\THC/generic/THCTensorMathBlas.cu:283 Error occurs, No graph saved Traceback (most recent call last): File "experiments.py", line 152, in main() File "experiments.py", line 46, in main evaluate(opts[''], opts[''], opts) File "experiments.py", line 79, in evaluate display_rewards=not options['--no-display']) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\trainer\evaluation.py", line 78, in init self.agent.set_writer(self.writer) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\deep_q_network\pytorch.py", line 100, in set_writer self.writer.add_graph(self.value_net, input_to_model=(model_input,)), File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\tensorboardX\writer.py", line 804, in add_graph self._get_file_writer().add_graph(graph(model, input_to_model, verbose, profile_with_cuda, kwargs)) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\tensorboardX\pytorch_graph.py", line 333, in graph raise e File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\tensorboardX\pytorch_graph.py", line 324, in graph trace = torch.jit.trace(model, args) File "D:\Anacoda\lib\site-packages\torch\jit__init__.py", line 875, in trace check_tolerance, _force_outplace, _module_class) File "D:\Anacoda\lib\site-packages\torch\jit__init.py", line 1027, in trace_module module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace) File "D:\Anacoda\lib\site-packages\torch\nn\modules\module.py", line 548, in call__ result = self._slow_forward(*input, *kwargs) File "D:\Anacoda\lib\site-packages\torch\nn\modules\module.py", line 534, in _slow_forward result = self.forward(input, kwargs) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\common\models.py", line 290, in forward ego_embeddedatt, = self.forward_attention(x) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\common\models.py", line 303, in forward_attention ego, others = self.ego_embedding(ego), self.others_embedding(others) File "D:\Anacoda\lib\site-packages\torch\nn\modules\module.py", line 548, in call result = self._slow_forward(*input, kwargs) File "D:\Anacoda\lib\site-packages\torch\nn\modules\module.py", line 534, in _slow_forward result = self.forward(*input, *kwargs) File "C:\Users\tengl\AppData\Roaming\Python\Python37\site-packages\rl_agents\agents\common\models.py", line 72, in forward x = self.activation(layer(x)) File "D:\Anacoda\lib\site-packages\torch\nn\modules\module.py", line 548, in call result = self._slow_forward(input, kwargs) File "D:\Anacoda\lib\site-packages\torch\nn\modules\module.py", line 534, in _slow_forward result = self.forward(*input, kwargs) File "D:\Anacoda\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward return F.linear(input, self.weight, self.bias) File "D:\Anacoda\lib\site-packages\torch\nn\functional.py", line 1612, in linear output = input.matmul(weight.t()) RuntimeError: size mismatch, m1: [1 x 5], m2: [7 x 64] at C:/cb/pytorch_1000000000000/work/aten/src\THC/generic/THCTensorMathBlas.cu:283** INFO: Finished writing results. You can upload them to the scoreboard via gym.upload('C:\Users\tengl\rl-agents\scripts\out\HighwayEnv\DQNAgent\run_20200629-114727_13208')

Thanks a lot! Best regards.

[eleurent]

Hi @narutoten520,

env_obs_attention.json is the proper environment configuration that you should be using with the ego_attention.json agent configuration. The error you get is caused by a recent regression in highway-env, presumably introduced by https://github.com/eleurent/highway-env/commit/42a939120952d4b15bb10c1daffd6315cd930eb0, where the observation variable was renamed to observation_type. I will push a fix.

Thanks a lot for your feeback!

[eleurent]

Please update the rl-agents repository to master, and it should be fixed (I just tested locally)

[narutoten520]

@eleurent The problem is solved. Thanks a lot for your guidance.