How to evaluate different Scene

[H-ChanY]

Hello,

I am currently trying to evaluate my model using your simulator, particularly in Scene A, B, and C, as I've encountered issues with Scene D. However, I've noticed a problem when editing evaluate_policy.py: it seems to create scenarios where necessary objects, like the red block required for a "grab the red block" task, are not visible in the environment. I suspect they might be obscured by other objects, such as sliders.

Could you please advise on how I might adjust the scene settings or the script to ensure that all required objects are visible and accessible in the environment? Any guidance or suggestions you could provide would be greatly appreciated.

Thank you very much for your assistance.

Best regards,

H-ChanY

[lukashermann]

I think the problem occurs when you convert the initial condition of a test sequence into the concrete object and scene positions in this line. An initial condition is a dictionary that specifies the scene state at the beginning of the 5 step evaluation sequence.

{'led': 1,
 'lightbulb': 1,
 'slider': 'left',
 'drawer': 'open',
 'red_block': 'slider_left',
 'blue_block': 'table',
 'pink_block': 'slider_right',
 'grasped': 0}

This gets converted to robot_obs and scene_obs which are then used to reset the environment state.

You can try to tweak the values in lines 228 to 232 that specify the exact block positions for scene D such that it works for the other scenes.

[lukashermann]

If you want to quickly test block positions, I think it is easiest if you run python calvin_env/calvin_env/envs/play_table_env.py scene=calvin_scene_C and you edit the values in the scene config here. By default, the positions get randomly sampled but you can also put fixed values as in this conf. Note that despite the name of the latter conf being calvin_scene_D_eval, we currently don't use this conf for the evaluation (it's legacy).

You can interact with the pybullet debug viewer and open the slider to see if the block is at the correct position. Probably it's enough if you change the x-dim of the values and leave y and z unchanged.