Error when loading the resume state

[RavenKiller]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: vx.x.x or master? master Habitat-Sim: vx.x.x or master? v2.4.0 Habitat is under active development, and we advise users to restrict themselves to stable releases of Habitat-Lab and Habitat-Sim. The bug you are about to report may already be fixed in the latest version.

Master branch contains 'bleeding edge' code, but we do appreciate bug reports for it!

🐛 Bug

I try to use habitat baselines to train a pointnav model. When interrupting a training process and resuming it, a missing key error occurs:

Error executing job with overrides: ['habitat_baselines.evaluate=False']
Traceback (most recent call last):
  File "/root/miniconda3/envs/h24/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/h24/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/.vscode-server/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/root/.vscode-server/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/root/.vscode-server/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/root/.vscode-server/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/root/.vscode-server/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/root/.vscode-server/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/run.py", line 77, in <module>
    main()
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/run.py", line 31, in main
    execute_exp(cfg, "eval" if cfg.habitat_baselines.evaluate else "train")
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/run.py", line 60, in execute_exp
    trainer.train()
  File "/root/miniconda3/envs/h24/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 619, in train
    self._init_train(resume_state)
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 243, in _init_train
    self._agent = self._create_agent(resume_state)
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 125, in _create_agent
    return baseline_registry.get_agent_access_mgr(
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/rl/ppo/single_agent_access_mgr.py", line 67, in __init__
    self._init_policy_and_updater(lr_schedule_fn, resume_state)
  File "/root/habitat-lab24/habitat-baselines/habitat_baselines/rl/ppo/single_agent_access_mgr.py", line 81, in _init_policy_and_updater
    self._updater.load_state_dict(resume_state["state_dict"])
  File "/root/miniconda3/envs/h24/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PPO:
        Missing key(s) in state_dict: "actor_critic.net.prev_action_embedding.weight", "actor_critic.net.tgt_embeding.weight", "actor_critic.net.tgt_embeding.bias", "actor_critic.net.visual_encoder.running_mean_and_var._mean", "actor_critic.net.visual_encoder.running_mean_and_var._var", "actor_critic.net.visual_encoder.running_mean_and_var._count", "actor_critic.net.visual_encoder.backbone.conv1.0.weight", "actor_critic.net.visual_encoder.backbone.conv1.1.weight", "actor_critic.net.visual_encoder.backbone.conv1.1.bias", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer1.0.downsample.0.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.downsample.1.weight", "actor_critic.net.visual_encoder.backbone.layer1.0.downsample.1.bias", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer1.1.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer1.2.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer2.0.downsample.0.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.downsample.1.weight", "actor_critic.net.visual_encoder.backbone.layer2.0.downsample.1.bias", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer2.1.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer2.2.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer2.3.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer3.0.downsample.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.downsample.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.0.downsample.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer3.1.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer3.2.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer3.3.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer3.4.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer3.5.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer4.0.downsample.0.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.downsample.1.weight", "actor_critic.net.visual_encoder.backbone.layer4.0.downsample.1.bias", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer4.1.convs.7.bias", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.0.weight", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.1.weight", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.1.bias", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.3.weight", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.4.weight", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.4.bias", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.6.weight", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.7.weight", "actor_critic.net.visual_encoder.backbone.layer4.2.convs.7.bias", "actor_critic.net.visual_encoder.compression.0.weight", "actor_critic.net.visual_encoder.compression.1.weight", "actor_critic.net.visual_encoder.compression.1.bias", "actor_critic.net.visual_fc.1.weight", "actor_critic.net.visual_fc.1.bias", "actor_critic.net.state_encoder.rnn.weight_ih_l0", "actor_critic.net.state_encoder.rnn.weight_hh_l0", "actor_critic.net.state_encoder.rnn.bias_ih_l0", "actor_critic.net.state_encoder.rnn.bias_hh_l0", "actor_critic.net.state_encoder.rnn.weight_ih_l1", "actor_critic.net.state_encoder.rnn.weight_hh_l1", "actor_critic.net.state_encoder.rnn.bias_ih_l1", "actor_critic.net.state_encoder.rnn.bias_hh_l1", "actor_critic.action_distribution.linear.weight", "actor_critic.action_distribution.linear.bias", "actor_critic.critic.fc.weight", "actor_critic.critic.fc.bias". 
        Unexpected key(s) in state_dict: "net.prev_action_embedding.weight", "net.tgt_embeding.weight", "net.tgt_embeding.bias", "net.visual_encoder.running_mean_and_var._mean", "net.visual_encoder.running_mean_and_var._var", "net.visual_encoder.running_mean_and_var._count", "net.visual_encoder.backbone.conv1.0.weight", "net.visual_encoder.backbone.conv1.1.weight", "net.visual_encoder.backbone.conv1.1.bias", "net.visual_encoder.backbone.layer1.0.convs.0.weight", "net.visual_encoder.backbone.layer1.0.convs.1.weight", "net.visual_encoder.backbone.layer1.0.convs.1.bias", "net.visual_encoder.backbone.layer1.0.convs.3.weight", "net.visual_encoder.backbone.layer1.0.convs.4.weight", "net.visual_encoder.backbone.layer1.0.convs.4.bias", "net.visual_encoder.backbone.layer1.0.convs.6.weight", "net.visual_encoder.backbone.layer1.0.convs.7.weight", "net.visual_encoder.backbone.layer1.0.convs.7.bias", "net.visual_encoder.backbone.layer1.0.downsample.0.weight", "net.visual_encoder.backbone.layer1.0.downsample.1.weight", "net.visual_encoder.backbone.layer1.0.downsample.1.bias", "net.visual_encoder.backbone.layer1.1.convs.0.weight", "net.visual_encoder.backbone.layer1.1.convs.1.weight", "net.visual_encoder.backbone.layer1.1.convs.1.bias", "net.visual_encoder.backbone.layer1.1.convs.3.weight", "net.visual_encoder.backbone.layer1.1.convs.4.weight", "net.visual_encoder.backbone.layer1.1.convs.4.bias", "net.visual_encoder.backbone.layer1.1.convs.6.weight", "net.visual_encoder.backbone.layer1.1.convs.7.weight", "net.visual_encoder.backbone.layer1.1.convs.7.bias", "net.visual_encoder.backbone.layer1.2.convs.0.weight", "net.visual_encoder.backbone.layer1.2.convs.1.weight", "net.visual_encoder.backbone.layer1.2.convs.1.bias", "net.visual_encoder.backbone.layer1.2.convs.3.weight", "net.visual_encoder.backbone.layer1.2.convs.4.weight", "net.visual_encoder.backbone.layer1.2.convs.4.bias", "net.visual_encoder.backbone.layer1.2.convs.6.weight", "net.visual_encoder.backbone.layer1.2.convs.7.weight", "net.visual_encoder.backbone.layer1.2.convs.7.bias", "net.visual_encoder.backbone.layer2.0.convs.0.weight", "net.visual_encoder.backbone.layer2.0.convs.1.weight", "net.visual_encoder.backbone.layer2.0.convs.1.bias", "net.visual_encoder.backbone.layer2.0.convs.3.weight", "net.visual_encoder.backbone.layer2.0.convs.4.weight", "net.visual_encoder.backbone.layer2.0.convs.4.bias", "net.visual_encoder.backbone.layer2.0.convs.6.weight", "net.visual_encoder.backbone.layer2.0.convs.7.weight", "net.visual_encoder.backbone.layer2.0.convs.7.bias", "net.visual_encoder.backbone.layer2.0.downsample.0.weight", "net.visual_encoder.backbone.layer2.0.downsample.1.weight", "net.visual_encoder.backbone.layer2.0.downsample.1.bias", "net.visual_encoder.backbone.layer2.1.convs.0.weight", "net.visual_encoder.backbone.layer2.1.convs.1.weight", "net.visual_encoder.backbone.layer2.1.convs.1.bias", "net.visual_encoder.backbone.layer2.1.convs.3.weight", "net.visual_encoder.backbone.layer2.1.convs.4.weight", "net.visual_encoder.backbone.layer2.1.convs.4.bias", "net.visual_encoder.backbone.layer2.1.convs.6.weight", "net.visual_encoder.backbone.layer2.1.convs.7.weight", "net.visual_encoder.backbone.layer2.1.convs.7.bias", "net.visual_encoder.backbone.layer2.2.convs.0.weight", "net.visual_encoder.backbone.layer2.2.convs.1.weight", "net.visual_encoder.backbone.layer2.2.convs.1.bias", "net.visual_encoder.backbone.layer2.2.convs.3.weight", "net.visual_encoder.backbone.layer2.2.convs.4.weight", "net.visual_encoder.backbone.layer2.2.convs.4.bias", "net.visual_encoder.backbone.layer2.2.convs.6.weight", "net.visual_encoder.backbone.layer2.2.convs.7.weight", "net.visual_encoder.backbone.layer2.2.convs.7.bias", "net.visual_encoder.backbone.layer2.3.convs.0.weight", "net.visual_encoder.backbone.layer2.3.convs.1.weight", "net.visual_encoder.backbone.layer2.3.convs.1.bias", "net.visual_encoder.backbone.layer2.3.convs.3.weight", "net.visual_encoder.backbone.layer2.3.convs.4.weight", "net.visual_encoder.backbone.layer2.3.convs.4.bias", "net.visual_encoder.backbone.layer2.3.convs.6.weight", "net.visual_encoder.backbone.layer2.3.convs.7.weight", "net.visual_encoder.backbone.layer2.3.convs.7.bias", "net.visual_encoder.backbone.layer3.0.convs.0.weight", "net.visual_encoder.backbone.layer3.0.convs.1.weight", "net.visual_encoder.backbone.layer3.0.convs.1.bias", "net.visual_encoder.backbone.layer3.0.convs.3.weight", "net.visual_encoder.backbone.layer3.0.convs.4.weight", "net.visual_encoder.backbone.layer3.0.convs.4.bias", "net.visual_encoder.backbone.layer3.0.convs.6.weight", "net.visual_encoder.backbone.layer3.0.convs.7.weight", "net.visual_encoder.backbone.layer3.0.convs.7.bias", "net.visual_encoder.backbone.layer3.0.downsample.0.weight", "net.visual_encoder.backbone.layer3.0.downsample.1.weight", "net.visual_encoder.backbone.layer3.0.downsample.1.bias", "net.visual_encoder.backbone.layer3.1.convs.0.weight", "net.visual_encoder.backbone.layer3.1.convs.1.weight", "net.visual_encoder.backbone.layer3.1.convs.1.bias", "net.visual_encoder.backbone.layer3.1.convs.3.weight", "net.visual_encoder.backbone.layer3.1.convs.4.weight", "net.visual_encoder.backbone.layer3.1.convs.4.bias", "net.visual_encoder.backbone.layer3.1.convs.6.weight", "net.visual_encoder.backbone.layer3.1.convs.7.weight", "net.visual_encoder.backbone.layer3.1.convs.7.bias", "net.visual_encoder.backbone.layer3.2.convs.0.weight", "net.visual_encoder.backbone.layer3.2.convs.1.weight", "net.visual_encoder.backbone.layer3.2.convs.1.bias", "net.visual_encoder.backbone.layer3.2.convs.3.weight", "net.visual_encoder.backbone.layer3.2.convs.4.weight", "net.visual_encoder.backbone.layer3.2.convs.4.bias", "net.visual_encoder.backbone.layer3.2.convs.6.weight", "net.visual_encoder.backbone.layer3.2.convs.7.weight", "net.visual_encoder.backbone.layer3.2.convs.7.bias", "net.visual_encoder.backbone.layer3.3.convs.0.weight", "net.visual_encoder.backbone.layer3.3.convs.1.weight", "net.visual_encoder.backbone.layer3.3.convs.1.bias", "net.visual_encoder.backbone.layer3.3.convs.3.weight", "net.visual_encoder.backbone.layer3.3.convs.4.weight", "net.visual_encoder.backbone.layer3.3.convs.4.bias", "net.visual_encoder.backbone.layer3.3.convs.6.weight", "net.visual_encoder.backbone.layer3.3.convs.7.weight", "net.visual_encoder.backbone.layer3.3.convs.7.bias", "net.visual_encoder.backbone.layer3.4.convs.0.weight", "net.visual_encoder.backbone.layer3.4.convs.1.weight", "net.visual_encoder.backbone.layer3.4.convs.1.bias", "net.visual_encoder.backbone.layer3.4.convs.3.weight", "net.visual_encoder.backbone.layer3.4.convs.4.weight", "net.visual_encoder.backbone.layer3.4.convs.4.bias", "net.visual_encoder.backbone.layer3.4.convs.6.weight", "net.visual_encoder.backbone.layer3.4.convs.7.weight", "net.visual_encoder.backbone.layer3.4.convs.7.bias", "net.visual_encoder.backbone.layer3.5.convs.0.weight", "net.visual_encoder.backbone.layer3.5.convs.1.weight", "net.visual_encoder.backbone.layer3.5.convs.1.bias", "net.visual_encoder.backbone.layer3.5.convs.3.weight", "net.visual_encoder.backbone.layer3.5.convs.4.weight", "net.visual_encoder.backbone.layer3.5.convs.4.bias", "net.visual_encoder.backbone.layer3.5.convs.6.weight", "net.visual_encoder.backbone.layer3.5.convs.7.weight", "net.visual_encoder.backbone.layer3.5.convs.7.bias", "net.visual_encoder.backbone.layer4.0.convs.0.weight", "net.visual_encoder.backbone.layer4.0.convs.1.weight", "net.visual_encoder.backbone.layer4.0.convs.1.bias", "net.visual_encoder.backbone.layer4.0.convs.3.weight", "net.visual_encoder.backbone.layer4.0.convs.4.weight", "net.visual_encoder.backbone.layer4.0.convs.4.bias", "net.visual_encoder.backbone.layer4.0.convs.6.weight", "net.visual_encoder.backbone.layer4.0.convs.7.weight", "net.visual_encoder.backbone.layer4.0.convs.7.bias", "net.visual_encoder.backbone.layer4.0.downsample.0.weight", "net.visual_encoder.backbone.layer4.0.downsample.1.weight", "net.visual_encoder.backbone.layer4.0.downsample.1.bias", "net.visual_encoder.backbone.layer4.1.convs.0.weight", "net.visual_encoder.backbone.layer4.1.convs.1.weight", "net.visual_encoder.backbone.layer4.1.convs.1.bias", "net.visual_encoder.backbone.layer4.1.convs.3.weight", "net.visual_encoder.backbone.layer4.1.convs.4.weight", "net.visual_encoder.backbone.layer4.1.convs.4.bias", "net.visual_encoder.backbone.layer4.1.convs.6.weight", "net.visual_encoder.backbone.layer4.1.convs.7.weight", "net.visual_encoder.backbone.layer4.1.convs.7.bias", "net.visual_encoder.backbone.layer4.2.convs.0.weight", "net.visual_encoder.backbone.layer4.2.convs.1.weight", "net.visual_encoder.backbone.layer4.2.convs.1.bias", "net.visual_encoder.backbone.layer4.2.convs.3.weight", "net.visual_encoder.backbone.layer4.2.convs.4.weight", "net.visual_encoder.backbone.layer4.2.convs.4.bias", "net.visual_encoder.backbone.layer4.2.convs.6.weight", "net.visual_encoder.backbone.layer4.2.convs.7.weight", "net.visual_encoder.backbone.layer4.2.convs.7.bias", "net.visual_encoder.compression.0.weight", "net.visual_encoder.compression.1.weight", "net.visual_encoder.compression.1.bias", "net.visual_fc.1.weight", "net.visual_fc.1.bias", "net.state_encoder.rnn.weight_ih_l0", "net.state_encoder.rnn.weight_hh_l0", "net.state_encoder.rnn.bias_ih_l0", "net.state_encoder.rnn.bias_hh_l0", "net.state_encoder.rnn.weight_ih_l1", "net.state_encoder.rnn.weight_hh_l1", "net.state_encoder.rnn.bias_ih_l1", "net.state_encoder.rnn.bias_hh_l1", "action_distribution.linear.weight", "action_distribution.linear.bias", "critic.fc.weight", "critic.fc.bias". 
Exception ignored in: <function VectorEnv.__del__ at 0x145ce5086040>
Traceback (most recent call last):
  File "/root/habitat-lab24/habitat-lab/habitat/core/vector_env.py", line 615, in __del__
  File "/root/habitat-lab24/habitat-lab/habitat/core/vector_env.py", line 473, in close
  File "/root/habitat-lab24/habitat-lab/habitat/core/vector_env.py", line 131, in __call__
  File "/root/habitat-lab24/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 63, in send
  File "/root/miniconda3/envs/h24/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
  File "/root/miniconda3/envs/h24/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
  File "/root/miniconda3/envs/h24/lib/python3.9/multiprocessing/connection.py", line 368, in _send
BrokenPipeError: [Errno 32] Broken pipe

Steps to Reproduce

Steps to reproduce the behavior:

1. Run a pointnav baseline training process: python -u -m habitat_baselines.run --config-name=pointnav/ddppo_pointnav.yaml habitat_baselines.evaluate=False
2. Interrupt it after .habitat-resume-state.pth stored under the checkpoint folder.
3. Re-run the training process: python -u -m habitat_baselines.run --config-name=pointnav/ddppo_pointnav.yaml habitat_baselines.evaluate=False

Please note that without a minimal working example to reproduce the bug, we may not be able to help you.

Expected behavior

The program should load the resume state successfully.

Additional context

[Jackson-Crandell]

I had this same issue at first because ppo and ddppo use the same save directories (i.e. data/new_checkpoints). Changing the save directory allowed me to train a ddppo agent.