TypeError: must be real number, not NoneType

[seominseok00]

import gymnasium as gym

#@title Prepare environment, agent, and evaluation process.

NUM_EPISODES = 100  #@param {type: "integer"}

import sys
sys.path.append('C:/Users/Seominseok/content/RLAgents')

from rl_agents.trainer.evaluation import Evaluation
from rl_agents.agents.common.factory import load_agent, load_environment

%cd C:\Users\Seominseok\content\RLAgents\scripts
env_config = 'configs/IntersectionEnv/env.json'
agent_config = 'configs/IntersectionEnv/agents/DQNAgent/ego_attention_2h.json'

env = load_environment(env_config)
agent = load_agent(agent_config, env)
evaluation = Evaluation(env, agent, num_episodes=NUM_EPISODES, display_env=False, display_agent=False)
print(f"Ready to train {agent} on {env}")

evaluation.train()

#@title Run the learned policy for a few episodes.
env = load_environment(env_config)
env.config["offscreen_rendering"] = True
agent = load_agent(agent_config, env)
evaluation = Evaluation(env, agent, num_episodes=1)
evaluation.train()
print(evaluation.run_directory)

I ran the colab code you provided on my local computer, and everything seems to work fine up to evaluation.train().

test.ipynb

However, I encountered the following error in the code when attempting to test it.

TypeError                                 Traceback (most recent call last)
c:\Users\Seominseok\test\test.ipynb 셀 4 line 6
      4 agent = load_agent(agent_config, env)
      5 evaluation = Evaluation(env, agent, num_episodes=1)
----> 6 evaluation.train()
      7 print(evaluation.run_directory)

File C:\Users/Seominseok/test/RLAgents\rl_agents\trainer\evaluation.py:120, in Evaluation.train(self)
    118     self.run_batched_episodes()
    119 else:
--> 120     self.run_episodes()
    121 self.close()

File C:\Users/Seominseok/test/RLAgents\rl_agents\trainer\evaluation.py:148, in Evaluation.run_episodes(self)
    145 start_time = time.time()
    146 while not terminal:
    147     # Step until a terminal step is reached
--> 148     reward, terminal = self.step()
    149     rewards.append(reward)
    151     # Catch interruptions

File C:\Users/Seominseok/test/RLAgents\rl_agents\trainer\evaluation.py:180, in Evaluation.step(self)
    178 # Step the environment
    179 previous_observation, action = self.observation, actions[0]
...
     93         '-i', audiofile,
     94         '-acodec', 'copy'
     95     ])

TypeError: must be real number, not NoneType

I have uploaded my local environment to my repository.

[seominseok00]

Additionally, I'm encountering error in highway-env/scripts/utils.py where display = Display(visible=0, size=(1400, 900)) is defined.

Upon researching, it seems that pyvirtualdisplay does not support Windows. Can I still use the show_videos method on Windows?

pyvirtualdisplay: FileNotFoundError: [WinError 2] The system cannot find the specified

---

I solved this problem by installing Ubuntu, but the NoneType issue remains even in Ubuntu.

[eleurent]

For your original error, I'm not sure what's going on, but I see

     93         '-i', audiofile,
     94         '-acodec', 'copy'

So this may be related to the video recording. It works on colab, right? So it must be something about your local setup. Are you able to record videos with other gymnasium envs? Do you have ffmpeg installed?

About pyvirtualdisplay, I'm not sure this is supported on windows, I usually open the videos manually.

[seominseok00]

By referencing the script and making changes as follows, the TypeError persists, but the code runs successfully.

import gymnasium as gym
from gymnasium.wrappers import RecordVideo
import sys
import logging
from pprint import pprint
sys.path.append('/home/seominseok/content/highway-env')
sys.path.append('/home/seominseok/content/rl-agents')

import highway_env
highway_env.register_highway_envs()

import rl_agents.trainer.logger
from rl_agents.trainer.evaluation  import Evaluation
from rl_agents.agents.common.factory import load_agent, load_environment
from rl_agents.agents.common.graphics import AgentGraphics

logger = logging.getLogger(__name__)

TRAIN = False
NUM_EPISODES = 100

from os import chdir
chdir('/home/seominseok/content/rl-agents/scripts')

env_config = 'configs/IntersectionEnv/env.json'
agent_config = 'configs/IntersectionEnv/agents/DQNAgent/ego_attention_2h.json'

if __name__ == '__main__':
    env = load_environment(env_config)
    # env = gym.make("intersection-v0", render_mode='rgb_array')
    agent = load_agent(agent_config, env)

    if TRAIN:
        evaluation = Evaluation(env, agent, num_episodes=NUM_EPISODES, display_env=False, display_agent=False)
        evaluation.train()

    else:
        agent.load('/home/seominseok/content/rl-agents/scripts/out/IntersectionEnv/DQNAgent/run_20231001-185016_7367/checkpoint-final.tar')
        # agent.load('/Users/seominseok/content/rl-agents/scripts/out/IntersectionMergeEnv/DQNAgent/run_20231001-105124_5401/checkpoint-best.tar')
        evaluation = Evaluation(env, agent, num_episodes=NUM_EPISODES, display_env=False, display_agent=True)

        env = evaluation.env
        print('env configuration')
        pprint(env.default_config())
        env = RecordVideo(env, video_folder="/home/seominseok/content/videos", episode_trigger=lambda e: True)
        env.unwrapped.set_record_video_wrapper(env)
        env.configure({"simulation_frequency": 15})

        for videos in range(10):
            done = truncated = False
            obs, info = env.reset()
            while not(done or truncated):
                action = evaluation.agent.act(obs)
                osb, reward, done, truncated, info = env.step(action)
                env.render()
        env.close()

Additionally, I'm curious because in the IntersectionEnv, only three actions are considered.

I want to apply SocialAttentionDQN for five actions.

Which part should I modify? Should I only adjust the Model Network and Graphics?

[eleurent]

The number of action is defined by the environment's action space, see the docs.

In the IntersectionEnv, the default action space is https://highway-env.farama.org/actions/#discrete-meta-actions, but with only the longitudinal option enabled, which means the lane changing actions are not available, only SLOWER, IDLE and FASTER.

But LANE_LEFT and LANE_RIGHT would not do anything anyway because they only allow to change between lanes of the same road segment (same startpoint and endpoint), not to select the direction when a lane is branching out. For that, a new action type would need to be added.

[Firehdx]

import gymnasium as gym

#@title Prepare environment, agent, and evaluation process.

NUM_EPISODES = 100  #@param {type: "integer"}

import sys
sys.path.append('C:/Users/Seominseok/content/RLAgents')

from rl_agents.trainer.evaluation import Evaluation
from rl_agents.agents.common.factory import load_agent, load_environment

%cd C:\Users\Seominseok\content\RLAgents\scripts
env_config = 'configs/IntersectionEnv/env.json'
agent_config = 'configs/IntersectionEnv/agents/DQNAgent/ego_attention_2h.json'

env = load_environment(env_config)
agent = load_agent(agent_config, env)
evaluation = Evaluation(env, agent, num_episodes=NUM_EPISODES, display_env=False, display_agent=False)
print(f"Ready to train {agent} on {env}")

evaluation.train()

#@title Run the learned policy for a few episodes.
env = load_environment(env_config)
env.config["offscreen_rendering"] = True
agent = load_agent(agent_config, env)
evaluation = Evaluation(env, agent, num_episodes=1)
evaluation.train()
print(evaluation.run_directory)

I ran the colab code you provided on my local computer, and everything seems to work fine up to evaluation.train().

test.ipynb

However, I encountered the following error in the code when attempting to test it.

TypeError                                 Traceback (most recent call last)
c:\Users\Seominseok\test\test.ipynb 셀 4 line 6
      4 agent = load_agent(agent_config, env)
      5 evaluation = Evaluation(env, agent, num_episodes=1)
----> 6 evaluation.train()
      7 print(evaluation.run_directory)

File C:\Users/Seominseok/test/RLAgents\rl_agents\trainer\evaluation.py:120, in Evaluation.train(self)
    118     self.run_batched_episodes()
    119 else:
--> 120     self.run_episodes()
    121 self.close()

File C:\Users/Seominseok/test/RLAgents\rl_agents\trainer\evaluation.py:148, in Evaluation.run_episodes(self)
    145 start_time = time.time()
    146 while not terminal:
    147     # Step until a terminal step is reached
--> 148     reward, terminal = self.step()
    149     rewards.append(reward)
    151     # Catch interruptions

File C:\Users/Seominseok/test/RLAgents\rl_agents\trainer\evaluation.py:180, in Evaluation.step(self)
    178 # Step the environment
    179 previous_observation, action = self.observation, actions[0]
...
     93         '-i', audiofile,
     94         '-acodec', 'copy'
     95     ])

TypeError: must be real number, not NoneType

I have uploaded my local environment to my repository.

I met the same error; did you find a way to fix the NoneType issue?

[seominseok00]

@Firehdx Did you use git clone instead of pip install to clone highway-env and rl-agents?

Here is my directory setup:

content
├─📁 highway-env
└─📁 rl-agents

When I ran the code in the above setup, it worked properly.

import sys
import time
sys.path.append('/Users/seominseok/content/highway-env')
sys.path.append('/Users/seominseok/content/rl-agents')

import gymnasium as gym
from gymnasium.wrappers import RecordVideo
import logging
from pprint import pprint

import highway_env
highway_env.register_highway_envs()

from rl_agents.trainer.evaluation  import Evaluation
from rl_agents.agents.common.factory import load_agent, load_environment

logger = logging.getLogger(__name__)

TRAIN = True
NUM_EPISODES = 10000

from os import chdir
chdir('/Users/seominseok/content/rl-agents/scripts')

env_config = 'configs/IntersectiongeEnv/env.json'
agent_config = 'configs/IntersectionEnv/agents/DQNAgent/ego_attention_2h.json'

if __name__ == '__main__':
    start_time = time.time()

    env = load_environment(env_config)
    agent = load_agent(agent_config, env)

    if TRAIN:
        evaluation = Evaluation(env, agent, num_episodes=NUM_EPISODES, display_env=False, display_agent=False)
        evaluation.train()

    else:
        agent.load('C:/Users/Seominseok/content/rl-agents/scripts/out/IntersectionEnv/DQNAgent/run_20231012-163046_22216/checkpoint-best.tar')
        evaluation = Evaluation(env, agent, num_episodes=NUM_EPISODES, display_env=False, display_agent=True)

        env = evaluation.env
        print('env configuration')
        pprint(env.default_config())
        env = RecordVideo(env, video_folder="/Users/seominseok/content/videos", episode_trigger=lambda e: True)
        env.unwrapped.set_record_video_wrapper(env)
        env.configure({"simulation_frequency": 15})

        for videos in range(10):
            done = truncated = False
            obs, info = env.reset()
            while not(done or truncated):
                action = evaluation.agent.act(obs)
                osb, reward, done, truncated, info = env.step(action)
                env.render()
        env.close()

    end_time = time.time()

    # Convert seconds to hours, minutes, and seconds
    hours, remainder = divmod(end_time - start_time, 3600)
    minutes, seconds = divmod(remainder, 60)

    print(f"Program execution time: {int(hours)}h {int(minutes)}m {int(seconds)}s")

[Firehdx]

@seominseok00 I used pip, and I didn't know if it really matters. I encountered that issue when I tried to record a video of the env.

import gymnasium as gym
from gymnasium.wrappers import RecordVideo
from stable_baselines3 import DQN
import highway_env

env = gym.make("highway-fast-v0", render_mode="rgb_array")
model = DQN.load("highway_dqn/model", env=env)
env = RecordVideo(env, video_folder="highway_dqn/videos", episode_trigger=lambda e: True)
env.unwrapped.set_record_video_wrapper(env)
env.configure({"simulation_frequency": 15})

for videos in range(10):
    done = truncated = False
    obs, info = env.reset()
    while not (done or truncated):
        action, _states = model.predict(obs, deterministic=True)
        obs, reward, done, truncated, info = env.step(action)
        env.render()
env.close()

and the error went as follow:

TypeError                                 Traceback (most recent call last)
[d:\HDX\SJTU\](file:///D:/HDX/SJTU/)大三\AI\2023-SJTU-AI-HW\Project1\code\mycode\test.ipynb 单元格 8 line 2
     [22] action, _states = model.predict(obs, deterministic=True)
     [23] # Get reward
---> [24] obs, reward, done, truncated, info = env.step(action)
     [25] # Render
     [26]env.render()

File [d:\HDX\.conda\envs\highway\lib\site-packages\gymnasium\wrappers\record_video.py:191](file:///D:/HDX/.conda/envs/highway/lib/site-packages/gymnasium/wrappers/record_video.py:191), in RecordVideo.step(self, action)
    189 if not self.is_vector_env:
    190     if terminateds or truncateds:
--> 191         self.close_video_recorder()
    192 elif terminateds[0] or truncateds[0]:
    193     self.close_video_recorder()

File [d:\HDX\.conda\envs\highway\lib\site-packages\gymnasium\wrappers\record_video.py:204](file:///D:/HDX/.conda/envs/highway/lib/site-packages/gymnasium/wrappers/record_video.py:204), in RecordVideo.close_video_recorder(self)
    202 if self.recording:
    203     assert self.video_recorder is not None
--> 204     self.video_recorder.close()
    205 self.recording = False
    206 self.recorded_frames = 1

File [d:\HDX\.conda\envs\highway\lib\site-packages\gymnasium\wrappers\monitoring\video_recorder.py:157](file:///D:/HDX/.conda/envs/highway/lib/site-packages/gymnasium/wrappers/monitoring/video_recorder.py:157), in VideoRecorder.close(self)
    155     clip = ImageSequenceClip(self.recorded_frames, fps=self.frames_per_sec)
...
     93         '-i', audiofile,
     94         '-acodec', 'copy'
     95     ])

TypeError: must be real number, not NoneType

I tried the setup you mentioned above but it didn't work.

[seominseok00]

I encountered an error when using an RL agent.

It was fine when I did it with stable baselines.

How about trying it in a new Anaconda virtual environment?

[Firehdx]

I try it in a new env and encounter the same issue. Environment for reproduction:

python 3.9.18
gymnasium 0.29.1
highway-env 1.8.2
stable-baselines3 2.2.1

[Firehdx]

I pip the following pkg and the problem solved:

pip install pyvirtualdisplay

[seominseok00]

@Firehdx I'm glad it's resolved. Have a great day😊