liuzuxin
commented
2 years ago You are right, the goal environment is actually called button2 in the code. In the paper Sec. 4.1, there are descriptions of this task. The difference between the goal and the button task is whether there are gremlins (dynamic obstacles) in the environment. There are several reasons to do this when I ran the experiments:
- Better to control the randomness. The original
goaltask in SafetyGym is essentially doing the same thing as thebuttontask, except 1) the obstacle types and layout are different, and 2) whether the goal locations are fixed. The originalgoaltask will random sample goal locations and it is very hard to control its randomness in the code, which means that I may need to evaluate each agent for plenty of episodes to characterize the randomness for fair comparison. So instead, I just modify thebuttonenvironment, where the goal locations and the obstacle layouts could be easily fixed. - Reduce the training time. Also because of the randomness of the goal locations in the original
goaltask, it takes longer to train the agent. Since CVPO has an inner convex optimization (currently implemented via SciPy, will change it to PyTorch later), the training speed is actually very slow. So in order to save the training time, it is better to fix the randomness of the environment to make it converge faster. The comparison is still valid as long as all the agents are in the same set of tasks.
Please let me know if you are still confused! Thanks.
yardenas
greg3566
Which environment/task was used for the ICML 2022 paper: "Constrained Variational Policy Optimization for Safe Reinforcement Learning" (Liu, et al. 2022)? I am confused because script/goal.py contains a button environment, not a goal, in ENV_LIST.