Pytorch PPO Implementation, dimension difference

[kevin-mahon]

Hello, apologies if I do this wrong I don't contribute to open source often. I was attempting to run the Pytorch PPO implementation and kept getting several errors regarding the dimension of the observation 'o'. I believe this is because the return values from the environment are being mishandled. See my proposed changes below: Original:

o, ep_ret, ep_len = env.reset(), 0, 0

# Main loop: collect experience in env and update/log each epoch
for epoch in range(epochs):
    for t in range(local_steps_per_epoch):
        a, v, logp = ac.step(torch.as_tensor(o, dtype=torch.float32))

        next_o, r, d, _ = env.step(a)
        ep_ret += r
        ep_len += 1

        # save and log
        buf.store(o, a, r, v, logp)
        logger.store(VVals=v)

        # Update obs (critical!)
        o = next_o

        timeout = ep_len == max_ep_len
        terminal = d or timeout
        epoch_ended = t==local_steps_per_epoch-1

        if terminal or epoch_ended:
            if epoch_ended and not(terminal):
                print('Warning: trajectory cut off by epoch at %d steps.'%ep_len, flush=True)
            # if trajectory didn't reach terminal state, bootstrap value target
            if timeout or epoch_ended:
                _, v, _ = ac.step(torch.as_tensor(o, dtype=torch.float32))
            else:
                v = 0
            buf.finish_path(v)
            if terminal:
                # only save EpRet / EpLen if trajectory finished
                logger.store(EpRet=ep_ret, EpLen=ep_len)
            o, ep_ret, ep_len = env.reset(), 0, 0

To fix this I just get rid of the 'info' return value from env.reset, but I imagine this might not work for all environments (if they have extra returns for the env.step() for example).

Adjusted:

(o,_), ep_ret, ep_len = env.reset(), 0, 0

# Main loop: collect experience in env and update/log each epoch
for epoch in range(epochs):
    for t in range(local_steps_per_epoch):
        a, v, logp = ac.step(torch.as_tensor(o, dtype=torch.float32))

        next_o, r, d, _ = env.step(a)
        ep_ret += r
        ep_len += 1

        # save and log
        buf.store(o, a, r, v, logp)
        logger.store(VVals=v)

        # Update obs (critical!)
        o = next_o

        timeout = ep_len == max_ep_len
        terminal = d or timeout
        epoch_ended = t==local_steps_per_epoch-1

        if terminal or epoch_ended:
            if epoch_ended and not(terminal):
                print('Warning: trajectory cut off by epoch at %d steps.'%ep_len, flush=True)
            # if trajectory didn't reach terminal state, bootstrap value target
            if timeout or epoch_ended:
                _, v, _ = ac.step(torch.as_tensor(o, dtype=torch.float32))
            else:
                v = 0
            buf.finish_path(v)
            if terminal:
                # only save EpRet / EpLen if trajectory finished
                logger.store(EpRet=ep_ret, EpLen=ep_len)
            (o,_), ep_ret, ep_len = env.reset(), 0, 0

Let me know if anyone else has run into this problem or if this fix works.

[Alberto-Hache]

Hi, Kevin. I'm regularly using that same algorithm, pytorch PPO, and never had that problem. What error are you getting exactly? And on which environment(s)?

Best.

Alberto

El mié, 22 feb 2023, 21:25, kevin @.***> escribió:

Hello, apologies if I do this wrong I don't contribute to open source often. I was attempting to run the Pytorch PPO implementation and kept getting several errors regarding the dimension of the observation 'o'. I believe this is because the return values from the environment are being mishandled. See my proposed changes below: Original:

o, ep_ret, ep_len = env.reset(), 0, 0

Main loop: collect experience in env and update/log each epoch

for epoch in range(epochs): for t in range(local_steps_per_epoch): a, v, logp = ac.step(torch.as_tensor(o, dtype=torch.float32))

    next_o, r, d, _ = env.step(a)
    ep_ret += r
    ep_len += 1

    # save and log
    buf.store(o, a, r, v, logp)
    logger.store(VVals=v)

    # Update obs (critical!)
    o = next_o

    timeout = ep_len == max_ep_len
    terminal = d or timeout
    epoch_ended = t==local_steps_per_epoch-1

    if terminal or epoch_ended:
        if epoch_ended and not(terminal):
            print('Warning: trajectory cut off by epoch at %d steps.'%ep_len, flush=True)
        # if trajectory didn't reach terminal state, bootstrap value target
        if timeout or epoch_ended:
            _, v, _ = ac.step(torch.as_tensor(o, dtype=torch.float32))
        else:
            v = 0
        buf.finish_path(v)
        if terminal:
            # only save EpRet / EpLen if trajectory finished
            logger.store(EpRet=ep_ret, EpLen=ep_len)
        o, ep_ret, ep_len = env.reset(), 0, 0

To fix this I just get rid of the 'info' return value from env.reset, but I imagine this might not work for all environments (if they have extra returns for the env.step() for example).

Adjusted:

(o,_), ep_ret, ep_len = env.reset(), 0, 0

Main loop: collect experience in env and update/log each epoch

for epoch in range(epochs): for t in range(local_steps_per_epoch): a, v, logp = ac.step(torch.as_tensor(o, dtype=torch.float32))

    next_o, r, d, _ = env.step(a)
    print(f"next_o: {next_o}")
    print(f"r: {r}")
    print(f"d: {d}")
    print(f"_: {_}")
    ep_ret += r
    ep_len += 1

    # save and log
    buf.store(o, a, r, v, logp)
    logger.store(VVals=v)

    # Update obs (critical!)
    o = next_o

    timeout = ep_len == max_ep_len
    terminal = d or timeout
    epoch_ended = t==local_steps_per_epoch-1

    if terminal or epoch_ended:
        if epoch_ended and not(terminal):
            print('Warning: trajectory cut off by epoch at %d steps.'%ep_len, flush=True)
        # if trajectory didn't reach terminal state, bootstrap value target
        if timeout or epoch_ended:
            _, v, _ = ac.step(torch.as_tensor(o, dtype=torch.float32))
        else:
            v = 0
        buf.finish_path(v)
        if terminal:
            # only save EpRet / EpLen if trajectory finished
            logger.store(EpRet=ep_ret, EpLen=ep_len)
        (o,_), ep_ret, ep_len = env.reset(), 0, 0

Let me know if anyone else has run into this problem or if this fix works.

— Reply to this email directly, view it on GitHub https://github.com/openai/spinningup/issues/384, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRLIPR3LFAGJCMUDAKQDJDWYZY4VANCNFSM6AAAAAAVEZHQVM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[kevin-mahon]

Hello, I'm using a custom environment following the OpenAI Gym api. This would be the error (although this time for td3)

Traceback (most recent call last): File "src/laser_gym.py", line 43, in train_test() File "src/laser_gym.py", line 25, in train_test td3(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=1000, epochs=260, logger_kwargs=logger_kwargs) File "/home/kevin/work/spinningup/spinup/algos/pytorch/td3/td3.py", line 334, in td3 test_agent() File "/home/kevin/work/spinningup/spinup/algos/pytorch/td3/td3.py", line 276, in testagent o, r, d, = test_env.step(get_action(o, 0)) File "/home/kevin/work/spinningup/spinup/algos/pytorch/td3/td3.py", line 267, in get_action a = ac.act(torch.as_tensor(o, dtype=torch.float32)) ValueError: expected sequence of length 2 at dim 1 (got 1)

My reset and step functions are defined as follows:

def reset(self, options=None):
    self._agent_location = np.random.random_integers(int(self.LOW/2), int(self.HIGH/2), size=2).astype(self.DTYPE)
    obs = self._get_obs()
    info = self._get_info()
    return obs, info

def step(self, action):
    _new_agent_location = self._agent_location + np.matmul(self.response_matrix,action)
    rew = self.reward.get_reward(self._agent_location, action, _new_agent_location)
    self._agent_location = _new_agent_location

    if (np.abs(self._agent_location) > self.HIGH).any() or (np.abs(self._agent_location) < 0.05).all():
        done = True #hit the edge
    else:
        done = False 

    obs = self._get_obs()
    info = self._get_info()

    if self.render_mode == 'human':
        self.render()

    return obs, rew, done, info

I am a little confused how this could work for other people when it seems like all the environment's should return the same tuple (https://www.gymlibrary.dev/api/core/#gym.Env.reset)