search

eleurent / rl-agents

Implementations of Reinforcement Learning and Planning algorithms
MIT License
596 stars 153 forks source link

Issue with deepcopy #84

Open jlwu002 opened 2 years ago

jlwu002 commented 2 years ago

Hi, I find that I cannot perform deepcopy (from rl_agents.agents.common.factory import safe_deepcopy_env) of the environment when the observation type is "GrayscaleObservation". It works for the "Kinematics" observation type.

It says: TypeError: cannot pickle 'WorldSurface' object.

I wonder is there a way to fix it?

Thank you!

eleurent commented 2 years ago

The Grayscale observation relies on a pygame renderer to produce images of the scene, and pygame typically also interacts with I/O like displays and keyboard/mouse, which is probably why it cannot get copied.

A solution would be to edit the safe_deepcopy_env function in rl-agents (or the deepcopy method in highway-env) to copy all the proporties of the env object except for its attached pygame renderer used for observations. But then, you would not be able to generate observations from copies of the environment.

Which brings me to my question: why are you trying to copy the environment? I guess it is for doing planning, but if so you will not be using the observations anyway, so why not changing the observation type to something that does not use pygame (e.g. Kinematics observation)? Do you have a use case where you need both the copy and the image obs?

jlwu002 commented 2 years ago

I am doing something similar to MCTS, but the input to the NN is set to be an image. I need to have a copy of the environment so that I could explore different action paths from a state. Right now I set the seed, and keep a record of the action sequence so that I could simulate it from the very beginning to recover the path. That gets slow as the tree grows.

I am thinking, would it be possible to render the image outside of the environment? So that there is no "viewers" present when I do "safe_deepcopy_env"? (e.g., I set it to Kinematics observation, deepcopy the environment, and then do _render_to_grayscale outside in the rl-agents script somewhere.)

eleurent commented 2 years ago

Ah, I see. Yes, I agree that your proposal seems feasible: you should be able to copy the GrayscaleObservation class and have an instance created independently of the environment. Then, everytime you need to generate an observation, you can pass your (copied) environment instance to your GrayscaleObservation class, which can generate the image by accessing the environment state (road, vehicles, etc). as usual. Does that make sense?

jlwu002 commented 2 years ago

Yes. I'm trying that now. Thanks!

And thank you for the work!