Errors when attempting to train

feup-jmc commented 2 years ago

Hello and once again thanks for the amazing work!

Context

So far, I've managed to install the repo but running tests has proven difficult.

An issue I found early on was that running a sample training command like mentioned in the README didn't work as the program seemed to be unable to remember the environment passed by --env. This issue is mentioned in #1, and for the sake of consistency I will use the same command moving forward in these issues: $ python experiments/kitchen/dreamer/dreamer_v2_single_task_primitives.py --mode here_no_doodad --exp_prefix <> --env microwave

With a bit of debugging I think I've managed to solve it, although this issue is not limited to dreamer_v2_single_task_primitives.py : https://github.com/mihdalal/raps/blob/23c7d952c9d5e4db6fd96b9a56c1c0373d2bd121/rlkit/experiments/kitchen/dreamer/dreamer_v2_single_task_primitives.py#L117 should actually be search_space = {"env_name": [args.exp_prefix]}.

Using sed for this replacement shows the following to be all the affected files (I can make a PR later if it seems like this issue is not exclusive to me):

experiments/kitchen/dreamer/dreamer_v2_multi_task_primitives.py
experiments/kitchen/dreamer/dreamer_v2_raw_actions.py
experiments/kitchen/dreamer/dreamer_v2_single_task_primitives.py
experiments/kitchen/dreamer/dreamer_v2_vices.py
experiments/kitchen/ppo/ppo_raw_actions.py
experiments/kitchen/ppo/ppo_vices.py
experiments/metaworld/dreamer/dreamer_v2_single_task_primitives.py
experiments/metaworld/dreamer/dreamer_v2_single_task_raw_actions.py
experiments/metaworld/dreamer/dreamer_v2_single_task_vices.py

Issue

With this problem out of the way, however, here is the real focus of this issue - when running the command again after solving the aforementioned problem, this is the output error (with similar results for other kitchen training scripts):

$ python experiments/kitchen/dreamer/dreamer_v2_single_task_primitives.py --mode here_no_doodad --exp_prefix <> --env microwave                            
duplicate key ignored: env_name
2022-05-14 14:51:12.216333 WEST | Variant:
2022-05-14 14:51:12.216544 WEST | {
  "env_name": "microwave",
  "algorithm": "DreamerV2",
  "version": "normal",
  "replay_buffer_size": 500000,
  "algorithm_kwargs": {
    "num_epochs": 250,
    "num_eval_steps_per_epoch": 30,
    "min_num_steps_before_training": 2500,
    "num_pretrain_steps": 100,
    "max_path_length": 5,
    "batch_size": 417,
    "num_expl_steps_per_train_loop": 30,
    "num_train_loops_per_epoch": 40,
    "num_trains_per_train_loop": 10
  },
  "use_raw_actions": false,
  "env_suite": "kitchen",
  "pass_render_kwargs": true,
  "save_video": true,
  "env_kwargs": {
    "dense": false,
    "image_obs": true,
    "action_scale": 1.4,
    "use_workspace_limits": true,
    "control_mode": "primitives",
    "usage_kwargs": {
      "use_dm_backend": true,
      "use_raw_action_wrappers": false,
      "use_image_obs": true,
      "max_path_length": 5,
      "unflatten_images": false
    },
    "image_kwargs": {}
  },
  "actor_kwargs": {
    "discrete_continuous_dist": true,
    "init_std": 0.0,
    "num_layers": 4,
    "min_std": 0.1,
    "dist": "tanh_normal_dreamer_v1"
  },
  "vf_kwargs": {
    "num_layers": 3
  },
  "model_kwargs": {
    "model_hidden_size": 400,
    "stochastic_state_size": 50,
    "deterministic_state_size": 200,
    "embedding_size": 1024,
    "rssm_hidden_size": 200,
    "reward_num_layers": 2,
    "pred_discount_num_layers": 3,
    "gru_layer_norm": true,
    "std_act": "sigmoid2"
  },
  "trainer_kwargs": {
    "adam_eps": 1e-05,
    "discount": 0.8,
    "lam": 0.95,
    "forward_kl": false,
    "free_nats": 1.0,
    "pred_discount_loss_scale": 10.0,
    "kl_loss_scale": 0.0,
    "transition_loss_scale": 0.8,
    "actor_lr": 8e-05,
    "vf_lr": 8e-05,
    "world_model_lr": 0.0003,
    "reward_loss_scale": 2.0,
    "use_pred_discount": true,
    "policy_gradient_loss_scale": 1.0,
    "actor_entropy_loss_schedule": "1e-4",
    "target_update_period": 100,
    "detach_rewards": false,
    "imagination_horizon": 5
  },
  "num_expl_envs": 5,
  "num_eval_envs": 1,
  "expl_amount": 0.3,
  "seed": "94249",
  "exp_id": "0",
  "exp_prefix": "05-14-microwave",
  "instance_type": "None",
  "exp_name": "05-14-microwave_2022_05_14_14_51_12_0000--s-94249"
}
~/anaconda3/envs/IKEA_2/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
~/anaconda3/envs/IKEA_2/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
~/anaconda3/envs/IKEA_2/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
~/anaconda3/envs/IKEA_2/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
Process SpawnProcess-1:
Traceback (most recent call last):
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "~/xstar-raps/rlkit/rlkit/envs/mujoco_vec_wrappers.py", line 72, in _worker
    env = env_fn_wrapper.var()
  File "~/xstar-raps/rlkit/rlkit/torch/model_based/dreamer/experiments/kitchen_dreamer.py", line 42, in <lambda>
    lambda: primitives_make_env.make_env(env_suite, env_name, env_kwargs)
  File "~/xstar-raps/rlkit/rlkit/envs/primitives_make_env.py", line 129, in make_env
    env = make_base_kitchen_env(env_name, env_kwargs_new)
  File "~/xstar-raps/rlkit/rlkit/envs/primitives_make_env.py", line 99, in make_base_kitchen_env
    env = ALL_KITCHEN_ENVIRONMENTS[env_class](**env_kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/kitchen_envs.py", line 36, in __init__
    super(KitchenBase, self).__init__(**kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/franka/kitchen_multitask_v0.py", line 771, in __init__
    super(KitchenTaskRelaxV1, self).__init__(**kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/franka/kitchen_multitask_v0.py", line 216, in __init__
    distance=2.2, lookat=[-0.2, 0.5, 2.0], azimuth=70, elevation=-35
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/robot_env.py", line 82, in __init__
    model_path, frame_skip, camera_settings=camera_settings
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/mujoco_env.py", line 67, in __init__
    camera_settings=camera_settings,
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 70, in __init__
    self._patch_mjlib_accessors(self.model, self.sim.data)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 303, in _patch_mjlib_accessors
    data.site_xvelp = site_xvelp()
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 286, in site_xvelp
    jacp = site_jacp().reshape((model.nsite, 3, model.nv))
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 282, in site_jacp
    mjlib.mj_jacSite(model.ptr, data.ptr, jacp_view, None, i)
TypeError: jacp should be of shape (3, nv)
~/anaconda3/envs/IKEA_2/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
Process SpawnProcess-2:
Traceback (most recent call last):
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "~/xstar-raps/rlkit/rlkit/envs/mujoco_vec_wrappers.py", line 72, in _worker
    env = env_fn_wrapper.var()
  File "~/xstar-raps/rlkit/rlkit/torch/model_based/dreamer/experiments/kitchen_dreamer.py", line 42, in <lambda>
    lambda: primitives_make_env.make_env(env_suite, env_name, env_kwargs)
  File "~/xstar-raps/rlkit/rlkit/envs/primitives_make_env.py", line 129, in make_env
    env = make_base_kitchen_env(env_name, env_kwargs_new)
  File "~/xstar-raps/rlkit/rlkit/envs/primitives_make_env.py", line 99, in make_base_kitchen_env
    env = ALL_KITCHEN_ENVIRONMENTS[env_class](**env_kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/kitchen_envs.py", line 36, in __init__
    super(KitchenBase, self).__init__(**kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/franka/kitchen_multitask_v0.py", line 771, in __init__
    super(KitchenTaskRelaxV1, self).__init__(**kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/franka/kitchen_multitask_v0.py", line 216, in __init__
    distance=2.2, lookat=[-0.2, 0.5, 2.0], azimuth=70, elevation=-35
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/robot_env.py", line 82, in __init__
    model_path, frame_skip, camera_settings=camera_settings
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/mujoco_env.py", line 67, in __init__
    camera_settings=camera_settings,
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 70, in __init__
    self._patch_mjlib_accessors(self.model, self.sim.data)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 303, in _patch_mjlib_accessors
    data.site_xvelp = site_xvelp()
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 286, in site_xvelp
    jacp = site_jacp().reshape((model.nsite, 3, model.nv))
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 282, in site_jacp
    mjlib.mj_jacSite(model.ptr, data.ptr, jacp_view, None, i)
TypeError: jacp should be of shape (3, nv)
Process SpawnProcess-4:
Traceback (most recent call last):
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "~/xstar-raps/rlkit/rlkit/envs/mujoco_vec_wrappers.py", line 72, in _worker
    env = env_fn_wrapper.var()
  File "~/xstar-raps/rlkit/rlkit/torch/model_based/dreamer/experiments/kitchen_dreamer.py", line 42, in <lambda>
    lambda: primitives_make_env.make_env(env_suite, env_name, env_kwargs)
  File "~/xstar-raps/rlkit/rlkit/envs/primitives_make_env.py", line 129, in make_env
    env = make_base_kitchen_env(env_name, env_kwargs_new)
  File "~/xstar-raps/rlkit/rlkit/envs/primitives_make_env.py", line 99, in make_base_kitchen_env
    env = ALL_KITCHEN_ENVIRONMENTS[env_class](**env_kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/kitchen_envs.py", line 36, in __init__
    super(KitchenBase, self).__init__(**kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/franka/kitchen_multitask_v0.py", line 771, in __init__
    super(KitchenTaskRelaxV1, self).__init__(**kwargs)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/franka/kitchen_multitask_v0.py", line 216, in __init__
    distance=2.2, lookat=[-0.2, 0.5, 2.0], azimuth=70, elevation=-35
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/robot_env.py", line 82, in __init__
    model_path, frame_skip, camera_settings=camera_settings
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/mujoco_env.py", line 67, in __init__
    camera_settings=camera_settings,
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 70, in __init__
    self._patch_mjlib_accessors(self.model, self.sim.data)
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 303, in _patch_mjlib_accessors
    data.site_xvelp = site_xvelp()
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 286, in site_xvelp
    jacp = site_jacp().reshape((model.nsite, 3, model.nv))
  File "~/xstar-raps/d4rl/d4rl/kitchen/adept_envs/simulation/sim_robot.py", line 282, in site_jacp
    mjlib.mj_jacSite(model.ptr, data.ptr, jacp_view, None, i)
TypeError: jacp should be of shape (3, nv)
Traceback (most recent call last):
  File "experiments/kitchen/dreamer/dreamer_v2_single_task_primitives.py", line 139, in <module>
    exp_id=exp_id,
  File "~/xstar-raps/rlkit/rlkit/launchers/launcher_util.py", line 590, in run_experiment
    return run_experiment_here(method_call, **run_experiment_kwargs)
  File "~/xstar-raps/rlkit/rlkit/launchers/launcher_util.py", line 168, in run_experiment_here
    return experiment_function(variant)
  File "~/xstar-raps/rlkit/rlkit/torch/model_based/dreamer/experiments/kitchen_dreamer.py", line 45, in experiment
    expl_env = StableBaselinesVecEnv(env_fns=env_fns, start_method="fork")
  File "~/xstar-raps/rlkit/rlkit/envs/mujoco_vec_wrappers.py", line 134, in __init__
    observation_space, action_space = self.remotes[0].recv()
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "~/anaconda3/envs/IKEA_2/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

As I am not too well acquainted with the internals of MuJoCo, this would be very difficult for me to debug. I would very thankful if you could provide some insight in how to solve this error.

mihdalal commented 2 years ago

Hi, thanks for making this issue! As for the error regarding jacp - I have fixed it now, please pull from master. Everything should run just fine.

Re: should actually be search_space = {"env_name": [args.exp_prefix]} No that is incorrect. The code is actually correct as is. The env name should be set by args.env.

feup-jmc commented 2 years ago

Hello, Mr. Dalal

First of all, I must say I am extremely grateful for your much needed help in such an hour of need. I have merged with master and no longer do I get jacp errors. So far everything seems fine but I cannot definitively greenlight the changes just yet as my workstation is too busy to have enough memory to run RAPS (RuntimeError: CUDA error: out of memory) .

There is, however, something I must mention. Without the fixes to search_space mentioned above, I still get the error mentioned in #1. Does the script run fine on your side without them? If so, I find it very strange that python would have a problem with such a specific part of the program.

I will keep you updated as the situation develops - hopefully the changes to search_space won't have any negative side effects, but for now I can only hope for the best. 🤞

mihdalal commented 2 years ago

you can reduce the number of vec_envs to reduce GPU memory usage. The reason you get the error mentioned in #1 is that you aren't setting exp_prefix to reasonable value (<> is just a placeholder), replace it with a string, say test for example, then it will work fine. for example: python experiments/kitchen/dreamer/dreamer_v2_single_task_primitives.py --mode here_no_doodad --exp_prefix test --env microwave As I said above, don't change the search space code, your change is incorrect.

feup-jmc commented 2 years ago

You are right, replacing <> in field --exp_prefix does indeed solve the problem without the need to change the code! However, the use/behaviour of --exp_prefix is not made very clear in the README, so I would suggest changing it to make issues such as these less likely to happen in the future (what exactly does the field do? what is the value used for?).

That said, thank you for all the support this far! I am closing the issue as the underlying issue has been resolved.

mihdalal / raps

Errors when attempting to train #4

Context

Issue