search

rail-berkeley / rlkit

Collection of reinforcement learning algorithms
MIT License
2.45k stars 550 forks source link

Having SkewFit deal with stochastic environments #80

Closed edessa closed 4 years ago

edessa commented 4 years ago

Hi,

The environment I'm applying SkewFit over has changing background colors that change upon the environment's reset, meaning naively sampling goals from the VAE doesn't really work here. I changed the goal sampling mode in sawyer_push.py from "vae_prior" to "env" to make stochastic environments work, but it doesn't seem to ever start training. I've changed the number of eval/expl steps to be very low, but it seems to still never really start. Is directly sampling goal images from the environment not quite supported?

vitchyr commented 4 years ago

I recommend using presampled goals in that case. Sampling from the env is extremely slow when you use images (there should be a warning about it). The door example is an example. You'll need to save a bunch of images beforehand into a numpy file. Let me know if this makes sense and I can look into adding an example of this

edessa commented 4 years ago

Thanks for the response! I actually experimented exactly with what you said using the presampled goal methods you have in open door/pick and place, but the VAE reconstructions were very poor despite many things I tried, so I'm in the middle of swapping the online VAE implementation with a pre-trained loading. Not sure if it would be helpful for users to want to use a pre-trained VAE (it should at least be much more stable), but let me know if it's worth pushing to the repo.