search

minosworld / minos

MINOS: Multimodal Indoor Simulator
MIT License
201 stars 33 forks source link

Getting semantic output from simulator #134

Open jsdd25 opened 5 years ago

jsdd25 commented 5 years ago

Hi! There are some questions, I would like to ask about sensor semantic data. I have run the benchmark code which is released at https://github.com/minosworld/unreal. I noticed that the shape of the input images are [84, 84, 3], which are color frame. But, I want to input semantic frames into the model. Where can I obtain these semantic frames? I noticed that the step() function in RoomSimulator.py returns observations captured by the agent at the end of the step. response in step() function of RoomSimulator.py obviously returns observations captured by the agent. But there are no dict_keys: objectType and objectId in response. How can I obtain semantic frames? Thank you very much!

angelxuanchang commented 5 years ago

To get the semantic frames, you will need to turn those sensors on and hookup them up in the unreal code. You can turn them on by setting the appropriate observations to True and adding them to the outputs and modalities. You should make your own environment configurations (see https://github.com/minosworld/minos/blob/master/minos/config/envs for example environments and https://github.com/minosworld/minos/blob/master/minos/config/sim_config.py for the default simulator configuration)

jsdd25 commented 5 years ago

Thanks @angelxuanchang You said this: 'You can turn them on by setting the appropriate observations to True and adding them to the outputs and modalities' means setting 'objects' to True? I tried to set 'objects' in observations to True and and adding it to the outputs and modalities. But, there is no dict_key: objects in the output of the step() function. When I tried to set 'depth' in observations to True, I can get dict_key: depth, which represents the depth value of image. So, are there any other changes should be made in the environment configurations in order to get the semantic frames? Please help me out here. Thank you very much!

angelxuanchang commented 5 years ago

Sorry, the defaults was a bit out of date. Please see updated https://github.com/minosworld/minos/blob/master/minos/config/sim_config.py (it should be objectId and objectType).

jsdd25 commented 5 years ago

Thanks @angelxuanchang I have replaced sim_config.py with the updated one. However, when I started experiments, I encountered some problems. When I run the this command: python3 main.py --env_type indoor --env_name pointgoal_suncg_se, I encountered the following problem:

jin@jin-System-Product-Name:~/minos$ python3 main.py --env_type indoor --env_name pointgoal_suncg_se
/home/jin/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "main.py", line 232, in <module>
    tf.app.run()
  File "/home/jin/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "main.py", line 229, in main
    app.run()
  File "main.py", line 86, in run
    objective_size = Environment.get_objective_size(flags.env_type, flags.env_name)
  File "/home/jin/minos/environment/environment.py", line 69, in get_objective_size
    return indoor_environment.IndoorEnvironment.get_objective_size(env_name)
  File "/home/jin/minos/environment/indoor_environment.py", line 27, in get_objective_size
    simargs = sim_config.get(env_name)
  File "/home/jin/minos/minos/config/sim_config.py", line 89, in get
    s['arch_only'] = override_args.get('arch_only', None)
AttributeError: 'NoneType' object has no attribute 'get'

It seems the parameter: override_args in get() function from sim_config.py is not passed in.

When I run the this command: python3 -m minos.tools.pygame_client --empty_room, I also encountered some problems:

jin@jin-System-Product-Name:~/minos$ python3 -m minos.tools.pygame_client --empty_room
pygame 1.9.4
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
  File "/home/jin/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/jin/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jin/minos/minos/tools/pygame_client.py", line 555, in <module>
    main()
  File "/home/jin/minos/minos/tools/pygame_client.py", line 511, in main
    args = parse_sim_args(parser)
  File "/home/jin/minos/minos/config/sim_args.py", line 236, in parse_sim_args
    sim_args = sim_config.get(args.env_config, vars(args))
  File "/home/jin/minos/minos/config/sim_config.py", line 119, in get
    simargs['scene']['defaultModelFormat'] = 'obj' if simargs['scene']['dataset'] == 'p5dScene' else None
KeyError: 'dataset'

Please help me fix these problems! Thank you very much!

jsdd25 commented 5 years ago

Hi @angelxuanchang , I had to add (L89-90) in sim_config.py (minos/minos/config)

if override_args is None:
        override_args = {}

When I run the this command: python3 main.py --env_type indoor --env_name pointgoal_suncg_se, I encountered the following problem:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/jin/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/jin/anaconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "main.py", line 45, in train_function
    trainer.prepare()
  File "/home/jin/minos/train/trainer.py", line 82, in prepare
    thread_index=self.thread_index)
  File "/home/jin/minos/environment/environment.py", line 37, in create_environment
    return indoor_environment.IndoorEnvironment(env_name, env_args, thread_index)
  File "/home/jin/minos/environment/indoor_environment.py", line 52, in __init__
    self._sim_obs_space = self._sim.get_observation_space(simargs['outputs'])
  File "/home/jin/minos/minos/lib/RoomSimulator.py", line 292, in get_observation_space
    self.init()
  File "/home/jin/minos/minos/lib/RoomSimulator.py", line 157, in init
    return self.new_episode()
  File "/home/jin/minos/minos/lib/RoomSimulator.py", line 113, in new_episode
    config['scene'] = {'fullId': self.params['scene']['source'] + '.' + ep_settings['scene_id'],
KeyError: 'source'

Please help me out here. Many thanks once again.

angelxuanchang commented 5 years ago

Thanks for letting us know about this - we have fixed override_args to default to {} in sim_config.py Are you able to successfully get the semantic segmentation output?

jsdd25 commented 5 years ago

Hi @angelxuanchang , thanks for your help. When I run this command: python3 main.py --env_type indoor --env_name pointgoal_suncg_se, I still encountered some problems:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/jin/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/jin/anaconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "main.py", line 45, in train_function
    trainer.prepare()
  File "/home/jin/minos/train/trainer.py", line 82, in prepare
    thread_index=self.thread_index)
  File "/home/jin/minos/environment/environment.py", line 37, in create_environment
    return indoor_environment.IndoorEnvironment(env_name, env_args, thread_index)
  File "/home/jin/minos/environment/indoor_environment.py", line 52, in __init__
    self._sim_obs_space = self._sim.get_observation_space(simargs['outputs'])
  File "/home/jin/minos/minos/lib/RoomSimulator.py", line 300, in get_observation_space
    self.init()
  File "/home/jin/minos/minos/lib/RoomSimulator.py", line 165, in init
    return self.new_episode()
  File "/home/jin/minos/minos/lib/RoomSimulator.py", line 113, in new_episode
    config['scene'] = {'fullId': self.params['scene']['source'] + '.' + ep_settings['scene_id'],
KeyError: 'source'

Then, checking simserver.log I get:

2018-12-24 11:04:08,511 INFO b'Configuration { fileCache: { size: 50 },'
2018-12-24 11:04:08,511 INFO b'  imageCache: { size: 200 },'
2018-12-24 11:04:08,511 INFO b'  imageQueue: { concurrency: 4 },'
2018-12-24 11:04:08,511 INFO b"  base_url: '/home/jin/work/',"
2018-12-24 11:04:08,511 INFO b'  assets_url:'
2018-12-24 11:04:08,511 INFO b"   'file:///home/jin/minos/minos/server/node_modules/sstk/ssc/../server/static' }"
2018-12-24 11:04:08,653 INFO b'Waiting for client connection on port 20596'
2018-12-24 11:04:08,664 ERROR b'Cannot register unknown asset 2d3ds'
2018-12-24 11:04:08,664 INFO b'register ../server/static/data/matterport/matterport3d.json'
2018-12-24 11:04:08,666 INFO b'Registered asset group: mp3d'
2018-12-24 11:04:08,668 INFO b'Got 90 assets'
2018-12-24 11:04:08,708 INFO b'register ../server/static/data/suncg/suncg.planner5d.models.json'
2018-12-24 11:04:08,709 INFO b'Replace asset group p5d'
2018-12-24 11:04:08,709 INFO b'Registered asset group: p5d'
2018-12-24 11:04:08,762 INFO b'Got 2623 assets'
2018-12-24 11:04:09,359 INFO b'Loaded lightSpecs for 202 models.'
2018-12-24 11:04:09,359 INFO b'register ../server/static/data/suncg/suncg.planner5d.textures.json'
2018-12-24 11:04:09,360 INFO b'Replace asset group p5dTexture'
2018-12-24 11:04:09,360 INFO b'Registered asset group: p5dTexture'
2018-12-24 11:04:09,548 INFO b'Got 67072 assets'
2018-12-24 11:04:09,683 INFO b'Client WVBfYwbv71i6lMdeAAAA connected on port 20596'
2018-12-24 11:04:09,806 INFO b'THREE.WebGLRenderer 84'
2018-12-24 11:04:09,807 ERROR b'THREE.WebGLRenderer: WEBGL_depth_texture extension not supported.'
2018-12-24 11:04:09,807 ERROR b'THREE.WebGLRenderer: OES_texture_float_linear extension not supported.'
2018-12-24 11:04:09,807 ERROR b'THREE.WebGLRenderer: OES_texture_half_float extension not supported.'
2018-12-24 11:04:09,807 ERROR b'THREE.WebGLRenderer: OES_texture_half_float_linear extension not supported.'
2018-12-24 11:04:09,807 ERROR b'THREE.WebGLRenderer: OES_standard_derivatives extension not supported.'
2018-12-24 11:04:09,814 INFO b"Creating Simulator with options { simulator: 'room_simulator',"
2018-12-24 11:04:09,814 INFO b'  numSimulators: 1,'
2018-12-24 11:04:09,814 INFO b"  modalities: [ 'color', 'measurements' ],"
2018-12-24 11:04:09,814 INFO b"  outputs: [ 'color', 'measurements', 'rewards', 'terminals' ],"
2018-12-24 11:04:09,814 INFO b'  resolution: [ 84, 84 ],'
2018-12-24 11:04:09,814 INFO b'  frameSkip: 1,'
2018-12-24 11:04:09,815 INFO b"  goal: { position: 'random', radius: 0.25 },"
2018-12-24 11:04:09,815 INFO b"  scenesFile: '/home/jin/minos/minos/config/../data/scenes.multiroom.csv',"
2018-12-24 11:04:09,816 INFO b'  statesFile:'
2018-12-24 11:04:09,816 INFO b"   '/home/jin/minos/minos/config/../data/episode_states.suncg.csv.bz2',"
2018-12-24 11:04:09,816 INFO b"  roomtypesFile: '/home/jin/minos/minos/config/../data/roomTypes.suncg.csv',"
2018-12-24 11:04:09,816 INFO b'  numEpisodesPerRestart: 1000,'
2018-12-24 11:04:09,816 INFO b'  numEpisodesPerScene: 10,'
2018-12-24 11:04:09,816 INFO b'  maxStatesPerScene: 1,'
2018-12-24 11:04:09,816 INFO b'  episodesPerSceneTest: 1,'
2018-12-24 11:04:09,816 INFO b'  episodesPerSceneTrain: 10,'
2018-12-24 11:04:09,816 INFO b"  episodeSchedule: 'train',"
2018-12-24 11:04:09,816 INFO b"  nonserializable: [ 'measure_fun', 'scene_filter', 'episode_filter' ],"
2018-12-24 11:04:09,816 INFO b'  agent: { radialClearance: 0.2 },'
2018-12-24 11:04:09,816 INFO b'  objectiveSize: 4,'
2018-12-24 11:04:09,816 INFO b"  logdir: './new_summary/sim02',"
2018-12-24 11:04:09,816 INFO b"  id: 'sim02',"
2018-12-24 11:04:09,816 INFO b"  simPath: '/home/jin/minos/minos/lib/../',"
2018-12-24 11:04:09,816 INFO b"  nodeBaseUrl: '/home/jin/work/',"
2018-12-24 11:04:09,816 INFO b'  port: 20596,'
2018-12-24 11:04:09,816 INFO b'  audio:'
2018-12-24 11:04:09,816 INFO b'   { port: 40974,'
2018-12-24 11:04:09,816 INFO b"     datapath: 'data/wav',"
2018-12-24 11:04:09,816 INFO b"     wallpath: '/home/jin/work/suncg/wall' },"
2018-12-24 11:04:09,816 INFO b'  width: 84,'
2018-12-24 11:04:09,816 ERROR b'[sensors] colorEncoding rgba is set, overriding encoding gray'

Then I print self.params['scene'].keys(), it shows dict_keys(['arch_only', 'retexture', 'empty_room', 'dataset', 'replaceModels', 'defaultModelFormat', 'defaultSceneFormat']). It seems there is no key value: 'source' in self.params['scene'] in RoomSimulator.py. Please help me out here. Many thanks once again.

angelxuanchang commented 5 years ago

It looks like you are using a slightly older version of minos (maybe 0.5.x ). Please try updating to either the master or 0.6.x branch.