Mesh export error (what tag am I missing?)

hayoung-jeremy commented 8 months ago

summary

Mesh exporting failed

reproduction of the error

successfully run the quick start command to generate pointcloud.ply file.

tried Mesh export command causes following error :

python launch.py --config custom/threestudio-gaussiandreamer/configs/gaussiandreamer_mvdream.yaml --export --gpu 0 resume=outputs/gs-sds-mvdream/an_astronaut_wearing_a_blue_suit@20240329-022833/ckpts/last.ckpt system.exporter_type=mesh-exporter system.geometry.isosurface_method=mc-cpu system.geometry.isosurface_resolution=256

Import times for custom modules:
   0.0 seconds (IMPORT FAILED): custom/shap-e
   0.0 seconds: custom/threestudio-mvdream
   0.1 seconds: custom/threestudio-dreamcraft3D
   0.1 seconds: custom/threestudio-shap-e
   0.6 seconds: custom/threestudio-lrm
   1.0 seconds: custom/threestudio-gaussiandreamer

Traceback (most recent call last):
  File "/home/dreamer/threestudio/launch.py", line 301, in <module>
    main(args, extras)
  File "/home/dreamer/threestudio/launch.py", line 163, in main
    cfg = load_config(args.config, cli_args=extras, n_gpus=n_gpus)
  File "/home/dreamer/threestudio/threestudio/utils/config.py", line 113, in load_config
    scfg = parse_structured(ExperimentConfig, cfg)
  File "/home/dreamer/threestudio/threestudio/utils/config.py", line 127, in parse_structured
    scfg = OmegaConf.structured(fields(**cfg))
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 375, in __getitem__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 369, in __getitem__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 451, in _get_impl
    return self._resolve_with_default(
  File "/home/dreamer/.local/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 96, in _resolve_with_default
    raise MissingMandatoryValue("Missing mandatory value: $FULL_KEY")
omegaconf.errors.MissingMandatoryValue: Missing mandatory value: tag
    full_key: tag
    object_type=dict

cxh0519 commented 8 months ago

You might remove all system-related parameters and try again. See the example provided in threestudio-3dgs: python launch.py --config path/to/config --export --gpu 0 system.prompt_processor.prompt="a delicious hamburger" resume=path/to/last.ckpt

hayoung-jeremy commented 8 months ago

Thank you for the reply, but encountered another error :

command I've run :

python launch.py --config outputs/gs-sds-mvdream/a_blue_denim_jacket@20240401-051653/configs/parsed.yaml --export --gpu 0 resume=outputs/gs-sds-mvdream/a_blue_denim_jacket@20240401-051653/ckpts/last.ckpt system.prompt_processor.prompt="a blue denim jacket"

the error message :

Traceback (most recent call last):
  File "/home/dreamer/threestudio/launch.py", line 301, in <module>
    main(args, extras)
  File "/home/dreamer/threestudio/launch.py", line 259, in main
    trainer.predict(system, datamodule=dm, ckpt_path=cfg.resume)
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 864, in predict
    return call._call_and_handle_interrupt(
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 903, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path)
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
    results = self._run_stage()
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1028, in _run_stage
    return self.predict_loop.run()
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/loops/prediction_loop.py", line 130, in run
    return self.on_run_end()
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/loops/prediction_loop.py", line 202, in on_run_end
    results = self._on_predict_epoch_end()
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/loops/prediction_loop.py", line 368, in _on_predict_epoch_end
    call._call_lightning_module_hook(trainer, "on_predict_epoch_end")
  File "/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 157, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/home/dreamer/threestudio/threestudio/systems/base.py", line 332, in on_predict_epoch_end
    exporter_output: List[ExporterOutput] = self.exporter()
  File "/home/dreamer/threestudio/custom/threestudio-gaussiandreamer/geometry/exporter.py", line 35, in __call__
    mesh: Mesh = self.geometry.extract_mesh()
  File "/home/dreamer/threestudio/custom/threestudio-gaussiandreamer/geometry/gaussian_io.py", line 268, in extract_mesh
    occ = self.extract_fields(resolution).detach().cpu().numpy()
  File "/home/dreamer/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/dreamer/threestudio/custom/threestudio-gaussiandreamer/geometry/gaussian_io.py", line 247, in extract_fields
    w = gaussian_3d_coeff(
NameError: name 'gaussian_3d_coeff' is not defined

codyreading commented 7 months ago

Hey @cxh0519

I noticed this too, where did you get this code from? Looks like this function is missing.

codyreading commented 7 months ago

Hey @cxh0519 and @hayoung-jeremy

It was just a missing function from DreamGaussian, I was able to fix that here: https://github.com/cxh0519/threestudio-gaussiandreamer/pull/8

hayoung-jeremy commented 7 months ago

Thank you for the reply @codyreading , below is the mesh export command after applying your pull request, and the result message :

python launch.py --config outputs/gs-sds-mvdream/a_blue_denim_jacket@20240412-015957/configs/parsed.yaml --export --gpu 0 resume=outputs/gs-sds-mvdream/a_blue_denim_jacket@20240412-015957/ckpts/last.ckpt system.exporter_type=mesh-exporter system.exporter.context_type=cuda

the result message :

[INFO] Loaded Multiview Diffusion!
[INFO] Using prompt [a blue denim jacket] and negative prompt [ugly, bad anatomy, blurry, pixelated obscure, unnatural colors, poor lighting, dull, and unclear, cropped, lowres, low quality, artifacts, duplicate, morbid, mutilated, poorly drawn face, deformed, dehydrated, bad proportions, unfocused]
[INFO] Using view-dependent prompts [side]:[a blue denim jacket, side view] [front]:[a blue denim jacket, front view] [back]:[a blue denim jacket, back view] [overhead]:[a blue denim jacket, overhead view]
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[INFO] You are using a CUDA device ('NVIDIA A100-SXM4-40GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
[INFO] Restoring states from the checkpoint path at outputs/gs-sds-mvdream/a_blue_denim_jacket@20240412-015957/ckpts/last.ckpt
[INFO] Number of points at initialisation:334614
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[INFO] Loaded model weights from the checkpoint at outputs/gs-sds-mvdream/a_blue_denim_jacket@20240412-015957/ckpts/last.ckpt
/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'predict_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=29` in the `DataLoader` to improve performance.
Predicting DataLoader 0:   0%|                                                                                                                                                                                                                                            | 0/120 [00:00<?, ?it/s]/home/dreamer/.local/lib/python3.10/site-packages/pytorch_lightning/loops/prediction_loop.py:255: predict returned None if it was on purpose, ignore this warning...
Predicting DataLoa/home/dreamer/.local/lib/python3.10/site-packages/torch/functional.py:507: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3549.)0:07<00:00, 15.84it/s]  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
16it [00:12,  1.33it/s]
[INFO] mesh cleaning: (55622, 3) --> (39172, 3), (112100, 3) --> (79397, 3)
[INFO] marching cubes result: torch.Size([39172, 3]) (-0.6717594861984253-0.6467556357383728), torch.Size([79397, 3])
Predicting DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:25<00:00,  4.65it/s][INFO] Export assets saved to outputs/gs-sds-mvdream/a_blue_denim_jacket@20240412-015957/save

result

Thanks to @codyreading , I finally got the result .obj file, but no texture, no vertex color at all.
result :

codyreading commented 7 months ago

Hey @hayoung-jeremy

Yeah I took a look at the implementation, I don't think the textured mesh is implemented at all.

I believe it is a part of the DreamGaussian codebase so it can be brought over.

cxh0519 / threestudio-gaussiandreamer