OpenAI Gym Custom Environment Observation Space returns "None"

[serapergun]

After setting up a custom environment, I was testing whether my observation_space and action_space were properly defined. I was able to call:

• env.observation_space and get the properly defined observation_space

• env.observation_space.sample() and get a well-working sample

Though when calling env.observation_space.shape, I got "None" as a return which doesn't make sense considering I could sample it properly.

When I run: obs_space = env.observation_space action_space = env.action_space print("The observation space: {}".format(obs_space)) print("The action space: {}".format(action_space))

I got: The observation space: None The action space: None

Because of this error, I could not use these codes for example:

state_shape = env.observation_space.shape or state_shape= env.observation_space.n

I got en error: AttributeError: 'NoneType' object has no attribute 'shape'

In order to set my DQN, I need this.

What am I doing wrong?

Regards, Serap

[lionyouko]

You would like to call a env.reset() first before getting these attributes because they, in fact, are initiated as None. If you are planning to use stable_baselines3, for example, to choose the type of RL among the ones they have implemented, you can use check_env() function also. It can help you regarding these problems (sometimes you can have a wrong shape also, the check helps with it).

You can check in init() function that they are None by default. They will have a value only after the first communication between the simulation and the python process running the Env. That is because it needs to know what you want as actions and observations, and that comes from gymConnection module in omnet++ scenario.

When reset() is called, that first communication happens, and it fills observation and action as intended.

Please check if you are trying to print after or before a reset().

[dbuse]

Please update VeinsGym to 0.3.3+ and seee if the error persists. Also see https://github.com/tkn-tub/veins-gym/issues/8.

[serapergun]

@dbuse Thank you I updated.

@lionyouko Thanks for detailed help. I tried to use check_env() function but because of tensorflow.contrib error I could't make it. I reinstall tensorflow=1.14.0 (because of stable_baseline accepts), then my other codes broke down :(

But I could started to calculate and implement DQN. But, I think I got an error for the same reason. Here are my codes and error at the end:

`#!/usr/bin/env python3 """ Trivial example agent for the dcc environment. """

import logging import os import random import time

import gym import veins_gym from veins_gym import veinsgym_pb2

import torch, numpy as np from torch import nn

import numpy as np from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten from tensorflow.keras.optimizers import Adam import tensorflow

from rl.agents import DQNAgent from rl.policy import BoltzmannQPolicy from rl.memory import SequentialMemory

def serialize_action(actions): """Searialize a list of floats into an action.""" reply = veinsgym_pb2.Reply() reply.action.box.values.extend(actions) return reply.SerializeToString()

gym.register( id="veins-v1", entry_point="veins_gym:VeinsEnv", kwargs={ "scenario_dir": os.path.relpath( os.path.join( os.path.dirname(os.path.abspath(file)), "..", "scenario" ) ), "timeout": 150.0, "print_veins_stdout": False,

"action_serializer": serialize_action,

    # to run in a GUI, use:
    # "user_interface": "Qtenv",
    # "timeout": -1,
},

)

def build_model(states, actions): model = Sequential()

model.add(Flatten(input_shape=(1,states)))

model.add(Dense(24, activation='relu', input_shape=(1,)))
model.add(Dense(24, activation='relu'))
model.add(Dense(actions, activation='linear'))
#model.add(Flatten(input_shape=(states,1)))
return model

del model

def build_agent(model, actions): policy = BoltzmannQPolicy() memory = SequentialMemory(limit=50000, window_length=1) dqn = DQNAgent(model=model, memory=memory, policy=policy, nb_actions=actions, nb_steps_warmup=10, target_model_update=1e-2) return dqn

def main(): """ Run the trivial agent. """ logging.basicConfig(level=logging.DEBUG)

env = gym.make("veins-v1")
logging.info("Env created")

env.reset()
logging.info("Env reset")
done = False

episodes = 5
rewards = []

states = env.observation_space.shape
print(states)
actions = env.action_space.n
print(actions)

#del model 

model = build_model(states, actions)
model.summary()

print(model.input_shape)

dqn = build_agent(model, actions)
dqn.compile(Adam(lr=1e-3), metrics=['mae'])
dqn.fit(env, nb_steps=50000, visualize=False, verbose=1)

#agent.train(10000)

for episode in range(1, episodes+1):

    print("Env reset for episode:", episode)
    observation, reward, done, info = env.step(random.randint(0, 1))

    while not done:
        r_action = random.randint(0, 1)
        observation, reward, done, info = env.step(r_action)
        rewards.append(reward)

    # 1604022> the last action sent by omnetpp to veinsgym will be a shutdown
    # and that will make the done set to true. In the veinsgym code that this happens, it will
    # also generates a random of the observation (_parse request function)
    # that will be sent in step function
    # and must then be discarded (as it is an undesired step + 1 anyway)
    #if not done:
        logging.debug(
            "Last action: %s, Reward: %.3f, Observation: %s",
            r_action,
            reward,
            observation,

        )

    model = build_model(observation, r_action)
    model.summary()

    dqn = build_agent(model, r_action)
    dqn.compile(Adam(lr=1e-3), metrics=['mae'])
    dqn.fit(env, nb_steps=50000, visualize=False, verbose=1)   

    print(type(env.observation_space))
    print(type(env.action_space))

    print('Episode:{} Reward:{}'.format(episode, reward))

    print("Number of steps taken:", len(rewards))
    print("Mean reward:", sum(rewards) / len(rewards))
    rewards = []
    time.sleep(0.059)
    #episodes=episodes+1;

if name == "main": main() `

An the errors of Console as: 2022-06-27 16:31:21.593068: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/seergun/anaconda3/lib/python3.8/site-packages/cv2/../../lib64::/home/seergun/.mujoco/mujoco210/bin 2022-06-27 16:31:21.593098: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. INFO:root:Env created INFO:root:Closing VeinsEnv. DEBUG:root:Listening on random port 37635 DEBUG:root:Launching veins experiment using command['./run', '-uCmdenv', '-cGeneral', '--seed-set=0', '--.manager.seed=0', '--.gym_connection.port=37635']` DEBUG:root:Veins process launched with pid 985818 INFO:root:Launched veins experiment, waiting for request. INFO:root:Received first request from Veins, ready to run. (1,) 2 INFO:root:Env reset Model: "sequential"

---

Layer (type) Output Shape Param #

dense (Dense) (None, 24) 48

dense_1 (Dense) (None, 24) 600

dense_2 (Dense) (None, 2) 50

================================================================= Total params: 698 Trainable params: 698 Non-trainable params: 0

---

(None, 1) /home/seergun/anaconda3/lib/python3.8/site-packages/keras/optimizers/optimizer_v2/adam.py:110: UserWarning: The lr argument is deprecated, use learning_rate instead. super(Adam, self).init(name, *kwargs) 2022-06-27 16:31:25.093644: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/seergun/anaconda3/lib/python3.8/site-packages/cv2/../../lib64::/home/seergun/.mujoco/mujoco210/bin 2022-06-27 16:31:25.093683: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303) 2022-06-27 16:31:25.093704: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (seergun-OptiPlex-9010): /proc/driver/nvidia/version does not exist 2022-06-27 16:31:25.101442: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled Training for 50000 steps ... INFO:root:Closing VeinsEnv. DEBUG:root:Closing VeinsEnv server socket. DEBUG:root:Listening on random port 39673 DEBUG:root:Launching veins experiment using command `['./run', '-uCmdenv', '-cGeneral', '--seed-set=0', '--.manager.seed=0', '--*.gym_connection.port=39673'] DEBUG:root:Veins process launched with pid 985839 INFO:root:Launched veins experiment, waiting for request. INFO:root:Received first request from Veins, ready to run. Interval 1 (0 steps performed) Traceback (most recent call last): File "/home/seergun/Downloads/dcc-env-master/agents/dqnlearning.py", line 200, in <module> main() File "/home/seergun/Downloads/dcc-env-master/agents/dqnlearning.py", line 127, in main dqn.fit(env, nb_steps=50000, visualize=False, verbose=1) File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/core.py", line 168, in fit action = self.forward(observation) File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/agents/dqn.py", line 224, in forward q_values = self.compute_q_values(state) File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/agents/dqn.py", line 68, in compute_q_values q_values = self.compute_batch_q_values([state]).flatten() File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/agents/dqn.py", line 63, in compute_batch_q_values q_values = self.model.predict_on_batch(batch) File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py", line 1186, in predict_on_batch inputs, _, _ = self._standardize_user_data( File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py", line 2323, in _standardize_user_data return self._standardize_tensors( File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py", line 2351, in _standardize_tensors x = training_utils_v1.standardize_input_data( File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_utils_v1.py", line 633, in standardize_input_data raise ValueError('Error when checking ' + exception_prefix + ValueError: Error when checking input: expected dense_input to have 2 dimensions, but got array with shape (1, 1, 1)

Dimension and shape issues make trouble.

As different from these: Today, I tried totally different issue as: I tried to connect my project to veins-gym like you commented previous

But again :( unfortunately, got an error.

Maybe because of run file or snakemake I don't know :(

I really need help from you as experts. I'm badly stuck I am confused. What should I do? I tried to change serpentine-env and dcc-env to my project with different application layer files. Could not. I tried to connect my project to gym. Could not.

So sorry for the long issue. Regards,

[lionyouko]

@serapergun I think was you that was waiting for the example scenario, right? By this time your work should be much more advanced than it, but I going to let here a link for you: https://github.com/lionyouko/veinsgym_lowerloss.git

Please, @dbuse , could you see if it is suitable as an example for this repo? Please, see if you can run it as it is on a linux machine (maybe at maximum rebuild it in omnet). I don't know how to make snakemake files, so I still use omnet to build before to use, and the run script is solely to run the executable directly. Thank you