search

princeton-vl / infinigen

Infinite Photorealistic Worlds using Procedural Generation
https://infinigen.org
BSD 3-Clause "New" or "Revised" License
5.42k stars 470 forks source link

KeyError in core.py #370

Open yiyulics opened 2 weeks ago

yiyulics commented 2 weeks ago

Describe the bug

I was using Infinigen to generate a multi-view dataset on nature scenes. It keeps throwing KeyError in infinigen/infinigen/terrain/surface_kernel/core.py.

Steps to Reproduce

What version of the code were you using?

commit 4a7dc1a809b74a2a100798ac671ce26d0a6d29c3 (HEAD -> main, tag: v1.10.1, origin/main, origin/HEAD)

What command did you run?

/path/to/infinigen/infinigen/datagen/manage_jobs.py --output_folder /mnt/infinigen/mvs_nature_0 --num_scenes 30 --configs multiview_stereo.gin --pipeline_configs local_16GB.gin monocular.gin blender_gt.gin cuda_terrain.gin --overrides camera.spawn_camera_rigs.n_camera_rigs=30 compose_nature.animate_cameras_enabled=False --pipeline_overrides iterate_scene_tasks.n_camera_rigs=30

What are your FULL output logs?

crash_summaries.txt

11/08 11:08AM /mnt/infinigen/mvs_nature_0/29779a43/logs/fineterrain.err reason="KeyError:  'eroded'" node=None fatal=True
11/08 11:23AM /mnt/infinigen/mvs_nature_0/5d0dbbc2/logs/fineterrain.err reason="KeyError:  'eroded'" node=None fatal=True
11/08 11:44AM /mnt/infinigen/mvs_nature_0/7139391f/logs/fineterrain.err reason="KeyError:  'mountain_collection'" node=None fatal=True

29779a43/logs/fineterrain.err

/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
[11:06:55.341] [acceleratesupport] [INFO] | No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[11:06:55.611] [smb_client] [WARNING] | SMB_AUTH envvar is not set, smb_client upload will not work. Ignore this message if not using upload
[11:06:55.613] [init] [INFO] | Converted seed='29779a43' to scene_seed=695704131, parsed as hexadecimal
[11:06:55.623] [execute_tasks] [INFO] | infinigen version 1.10.1
[11:06:55.623] [execute_tasks] [INFO] | CUDA_VISIBLE_DEVICES=0
[11:06:55.623] [logging] [INFO] | [MAIN TOTAL]
[11:06:55.623] [logging] [INFO] | [Reading input blendfile]
[11:06:55.897] [logging] [INFO] | [Reading input blendfile] finished in 0:00:00.273506
[11:06:55.897] [blender] [WARNING] | Re-initialized 0 as trusted. Do not run infinigen on untrusted blend files. 
[11:06:55.897] [execute_tasks] [INFO] | Processing frames 1 through 48 inclusive
[11:06:55.959] [init] [INFO] | Available devices have types=['OPTIX', 'CUDA', 'CPU']
[11:06:55.959] [init] [INFO] | Cycles will use use_device_type='OPTIX', len(use_devices)=1
[11:06:55.967] [logging] [INFO] | [Create terrain]
[11:06:55.967] [core] [INFO] | Terrain using only on the fly on_the_fly_asset_folder=PosixPath('/mnt/infinigen/mvs_nature_0/29779a43/fine/assets')
[11:06:58.755] [core] [INFO] | Terrain elements: ['ground', 'landtiles', 'voronoi_rocks', 'liquid', 'atmosphere']
[11:06:58.755] [logging] [INFO] | [Create terrain] finished in 0:00:02.788670
[11:06:58.756] [core] [INFO] | attribute='ground_collection' will use material infinigen.assets.materials.mud
[11:06:58.756] [core] [INFO] | attribute='mountain_collection' will use material infinigen.assets.materials.sandstone
[11:06:58.756] [core] [INFO] | attribute='eroded' will use material infinigen.assets.materials.stone
[11:06:58.756] [core] [INFO] | attribute='liquid_collection' will use material infinigen.assets.materials.water
[11:06:58.756] [core] [INFO] | attribute='atmosphere' will use material infinigen.assets.materials.atmosphere_light_haze
[11:07:01.229] [logging] [INFO] | [meshing OpaqueTerrain]

  0%|          | 0/437501 [00:00<?, ?it/s]
 46%|████▌     | 201328/437501 [00:01<00:01, 121884.28it/s]
100%|██████████| 437501/437501 [00:03<00:00, 148625.61it/s]
100%|██████████| 437501/437501 [00:04<00:00, 101201.56it/s]

0it [00:00, ?it/s]
0it [00:00, ?it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 64860.37it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 93483.74it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 71575.15it/s]
[11:07:23.853] [logging] [INFO] | [compute attributes]
[11:07:23.853] [logging] [INFO] | [compute attributes] finished in 0:00:00.000078
[11:07:23.853] [logging] [INFO] | [concat meshes]
[11:07:23.853] [logging] [INFO] | [concat meshes] finished in 0:00:00.000086
[11:07:23.853] [logging] [INFO] | [meshing OpaqueTerrain] finished in 0:00:22.624186
[11:07:26.342] [logging] [INFO] | [meshing liquid]

  0%|          | 0/437501 [00:00<?, ?it/s]
 54%|█████▎    | 234882/437501 [00:01<00:00, 231803.94it/s]
100%|██████████| 437501/437501 [00:01<00:00, 226873.27it/s]
100%|██████████| 437501/437501 [00:02<00:00, 182983.98it/s]

0it [00:00, ?it/s]
0it [00:00, ?it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 3133.82it/s]
[11:07:47.067] [logging] [INFO] | [compute attributes]
[11:07:47.067] [logging] [INFO] | [compute attributes] finished in 0:00:00.000118
[11:07:47.067] [logging] [INFO] | [concat meshes]
[11:07:47.067] [logging] [INFO] | [concat meshes] finished in 0:00:00.000110
[11:07:47.067] [logging] [INFO] | [meshing liquid] finished in 0:00:20.724907
[11:07:49.555] [logging] [INFO] | [meshing atmosphere]

  0%|          | 0/437501 [00:00<?, ?it/s]
100%|██████████| 437501/437501 [00:00<00:00, 445609.14it/s]
100%|██████████| 437501/437501 [00:01<00:00, 277507.62it/s]

0it [00:00, ?it/s]
0it [00:00, ?it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 985.84it/s]
[11:08:09.433] [logging] [INFO] | [compute attributes]
[11:08:09.433] [logging] [INFO] | [compute attributes] finished in 0:00:00.000075
[11:08:09.433] [logging] [INFO] | [concat meshes]
[11:08:09.433] [logging] [INFO] | [concat meshes] finished in 0:00:00.000077
[11:08:09.433] [logging] [INFO] | [meshing atmosphere] finished in 0:00:19.878018
[11:08:09.457] [logging] [INFO] | [MAIN TOTAL] failed with <class 'KeyError'>
Traceback (most recent call last):
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/yiyu/infinigen/infinigen_examples/generate_nature.py", line 1040, in <module>
    main(args)
  File "/home/yiyu/infinigen/infinigen_examples/generate_nature.py", line 976, in main
    execute_tasks.main(
  File "/home/yiyu/infinigen/infinigen/core/execute_tasks.py", line 367, in main
    execute_tasks(
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/yiyu/infinigen/infinigen/core/execute_tasks.py", line 275, in execute_tasks
    terrain.fine_terrain(
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/yiyu/infinigen/infinigen/terrain/core.py", line 470, in fine_terrain
    fine_meshes, _ = self.export(mesher_backend=mesher_backend, cameras=cameras)
  File "/home/yiyu/infinigen/infinigen/terrain/core.py", line 322, in export
    surface_kernel(meshes_dict[mesh_name])
  File "/home/yiyu/infinigen/infinigen/terrain/surface_kernel/core.py", line 134, in __call__
    ret[var] *= params.vertex_attributes[self.attribute].reshape(
KeyError: 'eroded'
  In call to configurable 'fine_terrain' (<function Terrain.fine_terrain at 0x7ac6e2048940>)
  In call to configurable 'execute_tasks' (<function execute_tasks at 0x7ac6e1714a60>)

7139391f/logs/fineterrain.err

/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
[11:43:23.370] [acceleratesupport] [INFO] | No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[11:43:23.638] [smb_client] [WARNING] | SMB_AUTH envvar is not set, smb_client upload will not work. Ignore this message if not using upload
[11:43:23.640] [init] [INFO] | Converted seed='7139391f' to scene_seed=1899575583, parsed as hexadecimal
[11:43:23.650] [execute_tasks] [INFO] | infinigen version 1.10.1
[11:43:23.650] [execute_tasks] [INFO] | CUDA_VISIBLE_DEVICES=0
[11:43:23.651] [logging] [INFO] | [MAIN TOTAL]
[11:43:23.651] [logging] [INFO] | [Reading input blendfile]
[11:43:23.979] [logging] [INFO] | [Reading input blendfile] finished in 0:00:00.328386
[11:43:23.979] [blender] [WARNING] | Re-initialized 0 as trusted. Do not run infinigen on untrusted blend files. 
[11:43:23.979] [execute_tasks] [INFO] | Processing frames 1 through 48 inclusive
[11:43:24.046] [init] [INFO] | Available devices have types=['OPTIX', 'CUDA', 'CPU']
[11:43:24.046] [init] [INFO] | Cycles will use use_device_type='OPTIX', len(use_devices)=1
[11:43:24.055] [logging] [INFO] | [Create terrain]
[11:43:24.055] [core] [INFO] | Terrain using only on the fly on_the_fly_asset_folder=PosixPath('/mnt/infinigen/mvs_nature_0/7139391f/fine/assets')
[11:43:26.138] [core] [INFO] | Terrain elements: ['landtiles', 'voronoi_rocks', 'liquid', 'atmosphere']
[11:43:26.138] [logging] [INFO] | [Create terrain] finished in 0:00:02.083051
[11:43:26.139] [core] [INFO] | attribute='mountain_collection' will use material infinigen.assets.materials.mountain
[11:43:26.139] [core] [INFO] | attribute='liquid_collection' will use material infinigen.assets.materials.water
[11:43:26.139] [core] [INFO] | attribute='atmosphere' will use material infinigen.assets.materials.atmosphere_light_haze
[11:43:28.747] [logging] [INFO] | [meshing OpaqueTerrain]

  0%|          | 0/437501 [00:00<?, ?it/s]
 50%|████▉     | 218104/437501 [00:01<00:01, 150557.80it/s]
 95%|█████████▌| 417160/437501 [00:02<00:00, 155397.12it/s]
100%|██████████| 437501/437501 [00:03<00:00, 95636.69it/s] 
100%|██████████| 437501/437501 [00:03<00:00, 110939.38it/s]

0it [00:00, ?it/s]
0it [00:00, ?it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 85598.04it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 90654.99it/s]
[11:43:51.069] [logging] [INFO] | [compute attributes]
[11:43:51.069] [logging] [INFO] | [compute attributes] finished in 0:00:00.000078
[11:43:51.069] [logging] [INFO] | [concat meshes]
[11:43:51.069] [logging] [INFO] | [concat meshes] finished in 0:00:00.000081
[11:43:51.069] [logging] [INFO] | [meshing OpaqueTerrain] finished in 0:00:22.322756
[11:43:53.666] [logging] [INFO] | [meshing liquid]

  0%|          | 0/437501 [00:00<?, ?it/s]
 54%|█████▎    | 234882/437501 [00:00<00:00, 244332.57it/s]
100%|██████████| 437501/437501 [00:01<00:00, 234896.40it/s]
100%|██████████| 437501/437501 [00:02<00:00, 172640.90it/s]

0it [00:00, ?it/s]
0it [00:00, ?it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 55236.66it/s]
[11:44:14.587] [logging] [INFO] | [compute attributes]
[11:44:14.587] [logging] [INFO] | [compute attributes] finished in 0:00:00.000103
[11:44:14.588] [logging] [INFO] | [concat meshes]
[11:44:14.588] [logging] [INFO] | [concat meshes] finished in 0:00:00.000100
[11:44:14.588] [logging] [INFO] | [meshing liquid] finished in 0:00:20.921998
[11:44:17.185] [logging] [INFO] | [meshing atmosphere]

  0%|          | 0/437501 [00:00<?, ?it/s]
100%|██████████| 437501/437501 [00:00<00:00, 461744.34it/s]
100%|██████████| 437501/437501 [00:01<00:00, 281224.69it/s]

0it [00:00, ?it/s]
0it [00:00, ?it/s]

  0%|          | 0/15 [00:00<?, ?it/s]
100%|██████████| 15/15 [00:00<00:00, 72733.60it/s]
[11:44:37.138] [logging] [INFO] | [compute attributes]
[11:44:37.138] [logging] [INFO] | [compute attributes] finished in 0:00:00.000080
[11:44:37.138] [logging] [INFO] | [concat meshes]
[11:44:37.138] [logging] [INFO] | [concat meshes] finished in 0:00:00.000075
[11:44:37.138] [logging] [INFO] | [meshing atmosphere] finished in 0:00:19.953159
[11:44:37.219] [logging] [INFO] | [MAIN TOTAL] failed with <class 'KeyError'>
Traceback (most recent call last):
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/yiyu/infinigen/infinigen_examples/generate_nature.py", line 1040, in <module>
    main(args)
  File "/home/yiyu/infinigen/infinigen_examples/generate_nature.py", line 976, in main
    execute_tasks.main(
  File "/home/yiyu/infinigen/infinigen/core/execute_tasks.py", line 367, in main
    execute_tasks(
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/yiyu/infinigen/infinigen/core/execute_tasks.py", line 275, in execute_tasks
    terrain.fine_terrain(
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/yiyu/anaconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/yiyu/infinigen/infinigen/terrain/core.py", line 470, in fine_terrain
    fine_meshes, _ = self.export(mesher_backend=mesher_backend, cameras=cameras)
  File "/home/yiyu/infinigen/infinigen/terrain/core.py", line 322, in export
    surface_kernel(meshes_dict[mesh_name])
  File "/home/yiyu/infinigen/infinigen/terrain/surface_kernel/core.py", line 134, in __call__
    ret[var] *= params.vertex_attributes[self.attribute].reshape(
KeyError: 'mountain_collection'
  In call to configurable 'fine_terrain' (<function Terrain.fine_terrain at 0x7400e0650940>)
  In call to configurable 'execute_tasks' (<function execute_tasks at 0x7400dfceca60>)

Platform

Additional context

Could anyone kindly help me with this issue? Thanks.

yiyulics commented 4 days ago

Hi, @mazeyu. Do you have any updates on this issue? Thanks a lot.