Testing 2020 starter code

[dhruvbatra]

Several problems.

1. Install nvidia-docker. Note: only supports Linux; no Windows or MacOS.

Do we need nvidia-docker v1 or v2?

2. ./test_locally_objectnav_rgbd.sh --docker-name my_submission

Both testlocally*.sh files were not executable (needed chmod a+x).

3. Docker commands failed -- I suspect because of incorrect naming in submissions.sh

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied. See 'docker run --help'.

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ sudo ./test_locally_objectnav_rgbd.sh --docker-name my_submission docker: Error response from daemon: Unknown runtime specified nvidia. See 'docker run --help'.

[erikwijmans]

This looks like a permissions issue. There is an extra step needed to use docker without sudo: https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user

[dhruvbatra]

No, even with those steps, I am still seeing the same error. Also, I am running these commands with sudo.

[mathfac]

@dhruvbatra 1. 2 were fixed.

Regarding 3-rd one, looks like Docker installation issues. can you try these fixes.

[dhruvbatra]

Yes, 1 and 2 were indeed fixed.

3 is also fixed. Looks like I didn't have nvidia-docker2 installed. I was able to test pointnav local evaluation. Will test objectnav after MP3D download finishes.

However, now we have potentially a new problem:

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_pointnav_rgbd.sh --docker-name my_submission
2020-02-27 02:44:22,783 Initializing dataset PointNav-v1
2020-02-27 02:44:22,785 initializing sim Sim-v0
2020-02-27 02:44:30,852 Initializing task Nav-v0
2020-02-27 02:44:35,600 distance_to_goal: inf
2020-02-27 02:44:35,600 spl: 0.0

Is the distance_to_goal inf and spl of 0 expected for a randomagent?

[dhruvbatra]

Finally finished downloading the MP3D dataset. ObjectNav local tests failing. Are we sure the directory it's checking for is correct? Readme says habitat-challenge-data/mp3d/, code below is checking for habitat-challenge-data/scene_datasets/mp3d/.

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
2020-02-27 03:07:29,179 Initializing dataset ObjectNav-v1
2020-02-27 03:07:29,258 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0227 03:07:31.024206    18 ResourceManager.cpp:84] Cannot load from file habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb
E0227 03:07:31.028100    18 Simulator.cpp:106] cannot load habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb
Traceback (most recent call last):
  File "agent.py", line 26, in <module>
    main()
  File "agent.py", line 21, in main
    challenge = habitat.Challenge()
  File "/habitat-api/habitat/core/challenge.py", line 16, in __init__
    super().__init__(config_paths)
  File "/habitat-api/habitat/core/benchmark.py", line 30, in __init__
    self._env = Env(config=config_env)
  File "/habitat-api/habitat/core/env.py", line 96, in __init__
    id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR
  File "/habitat-api/habitat/sims/registration.py", line 19, in make_sim
    return _sim(**kwargs)
  File "/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 168, in __init__
    self._sim = habitat_sim.Simulator(self.sim_config)
  File "<attrs generated init habitat_sim.simulator.Simulator>", line 9, in __init__
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/habitat_sim-0.1.4-py3.6-linux-x86_64.egg/habitat_sim/simulator.py", line 64, in __attrs_post_init__
    self.reconfigure(config)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/habitat_sim-0.1.4-py3.6-linux-x86_64.egg/habitat_sim/simulator.py", line 177, in reconfigure
    self._config_backend(config)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/habitat_sim-0.1.4-py3.6-linux-x86_64.egg/habitat_sim/simulator.py", line 106, in _config_backend
    self._sim = hsim.SimulatorBackend(config.sim_cfg)
ValueError: Cannot load: habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb

[erikwijmans]

Is the distance_to_goal inf and spl of 0 expected for a randomagent?

No, not expected. @mathfac have we pushed out the new pointnav episodes? (If we haven't, then this is expected as the old episodes won't all be navigable under the new agent size).

[dhruvbatra]

Just tested again. Now pointnav local testing is also failing.

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_pointnav_rgbd.sh --docker-name my_submission
2020-03-01 00:25:58,158 Initializing dataset PointNav-v1
2020-03-01 00:25:58,164 initializing sim Sim-v0
Traceback (most recent call last):
  File "agent.py", line 26, in <module>
    main()
  File "agent.py", line 21, in main
    challenge = habitat.Challenge()
  File "/habitat-api/habitat/core/challenge.py", line 16, in __init__
    super().__init__(config_paths)
  File "/habitat-api/habitat/core/benchmark.py", line 30, in __init__
    self._env = Env(config=config_env)
  File "/habitat-api/habitat/core/env.py", line 96, in __init__
    id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR
  File "/habitat-api/habitat/sims/registration.py", line 19, in make_sim
    return _sim(**kwargs)
  File "/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 166, in __init__
    self.sim_config = self.create_sim_config(self._sensor_suite)
  File "/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 210, in create_sim_config
    )(self.config).get()
TypeError: 'NoneType' object is not callable

[ghost]

Just tested again. Now pointnav local testing is also failing.

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_pointnav_rgbd.sh --docker-name my_submission
2020-03-01 00:25:58,158 Initializing dataset PointNav-v1
2020-03-01 00:25:58,164 initializing sim Sim-v0
Traceback (most recent call last):
  File "agent.py", line 26, in <module>
    main()
  File "agent.py", line 21, in main
    challenge = habitat.Challenge()
  File "/habitat-api/habitat/core/challenge.py", line 16, in __init__
    super().__init__(config_paths)
  File "/habitat-api/habitat/core/benchmark.py", line 30, in __init__
    self._env = Env(config=config_env)
  File "/habitat-api/habitat/core/env.py", line 96, in __init__
    id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR
  File "/habitat-api/habitat/sims/registration.py", line 19, in make_sim
    return _sim(**kwargs)
  File "/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 166, in __init__
    self.sim_config = self.create_sim_config(self._sensor_suite)
  File "/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 210, in create_sim_config
    )(self.config).get()
TypeError: 'NoneType' object is not callable

I have the same issue

[ghost]

@erikwijmans @mathfac

[mathfac]

@AdventureO, @dhruvbatra should be fixed right now. The docker image fairembodied/habitat-challenge:2020 was rebuilt.

[dhruvbatra]

Confirming that this issue is solved in branch challenge-2020-evalai-testing.

I see output as expected for test_locally_pointnav_rgbd.sh.

I am still seeing a path problem for test_locally_objectnav_rgbd.sh but will open a new issue for that.