shuruiz
commented
5 years ago If you mean the problem itself, I will say the continuous version is more challenging. Because the action space in that version is continuous. Continuous control is more complex than discrete control. If you mean the difficulty to concur the problem with coding, I will say the MountainCar-v0 is harder. Because in gym environment settings, your action choices are only to push left/not push/push right. That is extremely hard to find an action sequence to achieve the goal because you cannot control your speed acceleration. In the continuous version, you can adjust your speed easily.
Actually, if you train a DQN model, which is the algorithm used in Alphago zero. From my personal experiments, it takes much much longer than traditional rl methods like sarsa(lambda). The explanation that I can come up with is that it is easy for deep neural nets to capture continuous variables but not discrete action sequence, because we usually do not sampling a long future trajectory due to the computation complexity and training time, for example, we usually only do a one-step return when calculating the discounted reward.
To over come this issue, I will suggest you use a longer trajectory (like, for your n-setp return, increase your n) when calculating you reward to lower your bias, though it increases your model variance.
Hope this helps.
samar1tan
sven1977
I've implemented Cross-Entropy method and passed CartPole-v0 and MountainCarContinuous-v0 with the same hyperparameters and default reward definition from gym. But it didn't work on MountainCar-v0 even with unwrapped version and much more episodes taken.