search

minosworld / minos

MINOS: Multimodal Indoor Simulator
MIT License
201 stars 33 forks source link

Evaluation does not follow the episode schedule in point_goal task / goal position not passed. #135

Closed mcimpoi closed 5 years ago

mcimpoi commented 5 years ago

Hi,

I tried several runs of the same agent (unreal), in the attempt to reproduce the results in Table2 in the paper, namely PoingGoal for Matterport3D small and medium.

However, looking at the logs, filtered by "EPISODE", for the first episode, at test time, the goal is at different coordinates ( was added manually ):

{
  'sceneId': 'mp3d.2t7WUuJeko7',
  'task': 'point_goal',
  'start': {
    'angle': 1.386,
    'position': [0.672, 0.551, 2.339],
    'cell': {
      'j': 39,
      'isValid': True,
      'i': 80,
      'id': 3707
    }
  },
  'goal': {
    'angle': 1.8403473282415732,

    **'position': [-4.813260249729765, 0.5735506993532181, 1.2931640217578368],**

    'room': 'R5',
    'cell': {
      'j': 29,
      'isValid': True,
      'i': 25,
      'id': 2722
    },
    'radius': 0.25
  },
  'shortestPath': {
    'distance': 7.601219330881972,
    'isValid': True,
    'doors': [],
    'rooms': [1, 5, 6]
  }
}

{
  'start': {
    'position': [0.672, 0.551, 2.339],
    'cell': {
      'id': 3707,
      'j': 39,
      'i': 80,
      'isValid': True
    },
    'angle': 1.386
  },
  'task': 'point_goal',
  'sceneId': 'mp3d.2t7WUuJeko7',
  'goal': {
    'radius': 0.25,

    **'position': [-1.235677195070037, 0.5514386199228466, -0.9918128308026164],**

    'cell': {
      'id': 619,
      'j': 6,
      'i': 61,
      'isValid': True
    },
    'room': 'R0',
    'angle': 0.08224559970336502
  },
  'shortestPath': {
    'distance': 4.262741699796953,
    'doors': [],
    'isValid': True,
    'rooms': [1]
  }
}

The goal coordinates are different for different runs, and also different from what the episode scheduler for test is providing: 

{
  "pathNumDoors": 0,
  "pathNumRooms": 3,
  "episode_id": 10714,
  "level": 0,
  "pathDoorIds": "",
  "dist": 5.209,
  "pathDist": 6.36,
  "goal": {
    "position": [-4.473, 0.029, 1.72],
    "id": ""
  },
  "scene_id": "2t7WUuJeko7",
  "room_id": "R5",
  "start": {
    "position": [0.672, 0.551, 2.339],
    "angle": 1.386
  }
}

Looking in RoomSimulator.py, seems the problem is in L96-103

        if 'goal' in ep_settings:
            if self.params['task'] == 'room_goal':
                room_id = ep_settings['room_id']
                config['goal'] = {'roomIds': [room_id]}
            else:
                object_id = ep_settings['goal']['id']
                if object_id:
                    config['goal'] = {'objectIds': [object_id]}

Which doesn't handle the point_goal case. If I try to set config['goal'] I get the simulator crash with the following stack trace:

 20:36:41,008 INFO b'set active level 0'
 20:36:41,015 ERROR b"Error fetching model-mp3d.2t7WUuJeko7-matterport-glb TypeError: Cannot read property 'position' of undefined"
 20:36:41,016 ERROR b'    at NavScene.getCellId (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:101809:42)'
 20:36:41,016 ERROR b'    at /research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:101988:21'
 20:36:41,016 ERROR b'    at arrayMap (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:4651:24)'
 20:36:41,016 ERROR b'    at Function.map (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:13568:15)'
 20:36:41,016 ERROR b'    at NavScene.computeShortestPath (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:101987:24)'
 20:36:41,016 ERROR b'    at NavScene.reset (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:101679:31)'
 20:36:41,016 ERROR b'    at SimState.reset (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:111700:24)'
 20:36:41,016 ERROR b'    at Simulator.__setSceneState (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:110007:15)'
 20:36:41,016 ERROR b'    at /research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:110243:12'
 20:36:41,017 ERROR b'    at wrappedCallback (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:30066:10)'
 20:36:41,017 ERROR b"Cannot load: TypeError: Cannot read property 'position' of undefined"

Any hints in how to fix this?

Thank you!

msavva commented 5 years ago

Hi @mcimpoi , Thank you for trying out MINOS and for reporting this bug. We have committed a fix to the master branch which should correctly handle point_goal and make goal specification handling a bit more robust overall. Please try running with the fix, and let us know if it the issue is resolved.

mcimpoi commented 5 years ago

Hi @msavva ,

Thank you for the quick fix! We were using the older version of the simulator (0.5.0); I upgraded -- the fix didn't seem to work with the previous version.

Running the UNREAL evaluation still doesn't seem to work with the fix -- the simulator hangs; then, checking simserver.log I get:

12:16:23,820 INFO b'Loading scene  mp3d.2t7WUuJeko7'
12:16:23,837 INFO b'preloading /work//mp3d/scans/2t7WUuJeko7/house_segmentations/2t7WUuJeko7.house using HouseLoader'
12:16:23,837 INFO b'preloading /work//mp3d/navgrid_10/2t7WUuJeko7/2t7WUuJeko7.furnished.grid.json'
12:16:24,102 INFO b'Removing subscribers for event drain [ 0 ]'
12:16:36,231 INFO b'GLTFLoader: 11986.273ms'
12:16:36,294 INFO b'Timing loadScene: 12474.004ms'
12:16:36,306 INFO b'Loaded mp3d.2t7WUuJeko7 bbdims: [9.254634141921997,3.4607720598578453,11.796436309814453]'
12:16:36,312 INFO b'Timing for initial is 0'
12:16:36,313 INFO b'Parsing grid for mp3d.2t7WUuJeko7 navmap'
12:16:36,315 INFO b'Refining grid'
12:16:36,315 INFO b'Populating occupancy'
12:16:36,326 INFO b'set active level 0'
12:16:36,333 ERROR b"Error fetching model-mp3d.2t7WUuJeko7-matterport-glb TypeError: Cannot read property 'position' of undefined"
12:16:36,333 ERROR b'    at NavScene.getCellId (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:96591:42)'
12:16:36,334 ERROR b'    at /research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:96770:21'
12:16:36,334 ERROR b'    at arrayMap (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:4651:24)'
12:16:36,334 ERROR b'    at Function.map (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:13568:15)'
12:16:36,334 ERROR b'    at NavScene.computeShortestPath (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:96769:24)'
12:16:36,334 ERROR b'    at NavScene.reset (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:96461:31)'
12:16:36,334 ERROR b'    at SimState.reset (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:106482:24)'
12:16:36,334 ERROR b'    at Simulator.__setSceneState (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:104789:15)'
12:16:36,334 ERROR b'    at /research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:105025:12'
12:16:36,334 ERROR b'    at wrappedCallback (/research/minos_world_forks/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:30066:10)'
12:16:36,334 ERROR b"Cannot load: TypeError: Cannot read property 'position' of undefined"

I had to add (L88-89) in sim_config.py (minos/minos/config)

if override_args is None:
       override_args =  {}

In order to get evaluate.py from minosworld/unreal repo (https://github.com/minosworld/unreal) to work.

Many thanks once again.

angelxuanchang commented 5 years ago

Thanks for letting us know about this - we have fixed override_args to default to {} in sim_config.py I'm closing this issue now - please let us know if you encounter other issues.

mcimpoi commented 5 years ago

Hi @angelxuanchang, @msavva, Might need reopening the issue.

I ran a simple test with the following code snippet -- trying to debug the same problem, and still did not get it running.

Here is what I tried (running version 9f3954207aeccabdc2df283b3706092e4466a9e9 / sstk@0.6.0 (github:smartscenes/sstk#3735e535dd4e7d7b006c197e14e9bdef356f28c8) ):

` from minos.config.sim_args import sim_config import minos.lib.common as common from minos.lib.Simulator import Simulator

simargs = sim_config.get('pointgoal_mp3d_s') sim = Simulator(simargs) sim.init()

get the first test episode

curr_schedule = 'test' ep_schedulers = common.create_episode_schedulers(simargs) test_ep_scheduler = ep_schedulers['test'] test_ep_scheduler.reset() ep_settings = test_ep_scheduler.next_episode()

create config

config = {} config['scene'] = {'fullId': simargs['scene']['dataset'] + '.' + ep_settings['scene_id'], 'level': ep_settings['level'], 'textureSet': curr_schedule} config['goal'] = {'type': 'position', 'position': ep_settings['goal']['position']} config['start'] = ep_settings['start']

sim.configure(config) result = sim.start() # And here are the problems :) `

I get lots of WARNING localhost:50319/socket.io [waiting for connection] HTTPConnectionPool(host='localhost', port=50319): Max retries exceeded with url: /socket.io/?t=1547218508808-0&EIO=3&transport=polling (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdd7a7d9550>: Failed to establish a new connection: [Errno 111] Connection refused',))

And in the logs I get this: 15:40:49,353 INFO b'Loading scene mp3d.gxdoqLR6rwA' 15:40:49,358 ERROR b'/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:108253' 15:40:49,358 ERROR b'\t if (this.opts.scene && this.opts.scene.fullId.startsWith("mp3d")) {' 15:40:49,358 ERROR b'\t ^' 15:40:49,358 ERROR b'' 15:40:49,358 ERROR b"TypeError: Cannot read property 'startsWith' of undefined" 15:40:49,358 ERROR b' at Simulator.__loadScene (/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:108253:50)' 15:40:49,358 ERROR b' at /minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:108205:14' 15:40:49,358 ERROR b' at /minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:108126:8' 15:40:49,358 ERROR b' at /minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:39080:17' 15:40:49,358 ERROR b' at next (/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:43936:30)' 15:40:49,358 ERROR b' at /minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:39576:17' 15:40:49,358 ERROR b' at Timeout._onTimeout (/minos/minos/server/node_modules/sstk/client/build/STK-core.bundle.js:108122:35)' 15:40:49,358 ERROR b' at ontimeout (timers.js:436:11)' 15:40:49,359 ERROR b' at tryOnTimeout (timers.js:300:5)' 15:40:49,359 ERROR b' at listOnTimeout (timers.js:263:5)' 15:40:49,370 INFO Finished simserver stderr 15:40:49,371 INFO Finished simserver stdout

Many thanks once again.

PS: Is there an easy way to debug SSTK related errors? (e.g. using the "source" file, not the generated code? sstk/client/build/STK-core.bundle.js:108122:35)

angelxuanchang commented 5 years ago

Thanks for reporting this issue. This should be now fixed.