Pareto Q-Learning fails with any environment that doesn't have 2D state/obs space

[LeonEricsson]

As the title suggests running PQL with an environment that doesn't have a 2D state/obs space will run into an error during training (pql.py):

state = int(np.ravel_multi_index(state, self.env_shape))

ravel_multi_index expects the first parameter to be of the same sequence length as the second otherwise it will throw a ValueError. Since env_shape is hard-coded to be 2D

self.env_shape = ( int(high_bound[0] - low_bound[0] + 1), int(high_bound[1] - low_bound[1] + 1), )

this assumes the state/obs is always of length 2. I assume something is off here, or that I've got something completely wrong but I'm consistently getting errors in environments that don't have a 2D obs space.

[ffelten]

Hi @LeonEricsson ,

You are right, our implementation PQL only works on deterministic environments with 2d observation, typically grid worlds.

I believe it can be extended to more dimensions with a bit of work though.

[LeonEricsson]

Hi @LeonEricsson ,

You are right, our implementation PQL only works on deterministic environments with 2d observation, typically grid worlds.

I believe it can be extended to more dimensions with a bit of work though.

I am able to run by just expanding the self.env_shape as

self.env_shape = tuple( int(high_bound[i] - low_bound[i] + 1) for i in range(len(low_bound)) )

but I am too unfamiliar with the algorithm/code to determine if this has further implications.

[ffelten]

@wilrop can you have a look at this? I think just changing the shape might indeed be enough?

[wilrop]

I will take a look at this next week after the ICLR deadline!

[ffelten]

@LeonEricsson this should be fixed now (I merged Willem's PR on main). Can you try it out? :-)

[ffelten]

Closing since inactive