Is multi GPU training available?

[hayoung-jeremy]

summary

• Is it able to run dreamcraft3D with the following instruction commented on threestudio's github README.md for Multi GPU training?
• I'm trying to run dreamcraft3D on ubuntu 22.04 with 4 A6000 GPUs(which is an instance from LambdaCloud).
• Installation via docker is successful
• running instruction command from threestudio's Multi GPU training section caused error.

reproduction of the error

1. Installation via docker on ubuntu22.04 was successful
2. running docker was also successful
3. gpu detection is available

4. error happens when running following command :

   python launch.py --config custom/threestudio-dreamcraft3D/configs/dreamcraft3d-coarse-nerf.yaml --gpu 0,1,2,3 --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path" data.batch_size=2 data.n_val_views=4

   the error message :

   
   dreamer@68374033e950:~/threestudio$ prompt="a delicious hamburger"
   dreamer@68374033e950:~/threestudio$ image_path="load/images/hamburger_rgba.png"
   dreamer@68374033e950:~/threestudio$ python launch.py --config custom/threestudio-dreamcraft3D/configs/dreamcraft3d-coarse-nerf.yaml --gpu 0,1,2,3 --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path" data.batch_size=2 data.n_val_views=4
   /home/dreamer/.local/lib/python3.10/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
     warnings.warn(
   /home/dreamer/.local/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
     return register_model(fn_wrapper)
   /home/dreamer/.local/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
     return register_model(fn_wrapper)
   /home/dreamer/.local/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
     return register_model(fn_wrapper)
   /home/dreamer/.local/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
     return register_model(fn_wrapper)
   /home/dreamer/.local/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
     return register_model(fn_wrapper)
   /home/dreamer/.local/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
     torch.utils._pytree._register_pytree_node(
   
   Import times for custom modules:
      0.3 seconds: custom/threestudio-dreamcraft3D
   
   [WARNING] Timestamp is disabled when using multiple GPUs, please make sure you have a unique tag.
   Seed set to 0
   Traceback (most recent call last):
     File "/home/dreamer/threestudio/launch.py", line 301, in <module>
       main(args, extras)
     File "/home/dreamer/threestudio/launch.py", line 168, in main
       dm = threestudio.find(cfg.data_type)(cfg.data)
     File "/home/dreamer/threestudio/custom/threestudio-dreamcraft3D/data/image.py", line 324, in __init__
       self.cfg = parse_structured(SingleImageDataModuleConfig, cfg)
     File "/home/dreamer/threestudio/threestudio/utils/config.py", line 127, in parse_structured
       scfg = OmegaConf.structured(fields(**cfg))
   TypeError: SingleImageDataModuleConfig.__init__() got an unexpected keyword argument 'n_val_views'```

[hayoung-jeremy]

I've checked SingleImageDataModuleConfig class from image.py, and there is no n_val_views declared. It seems not availble for Multi GPU training, right?

class SingleImageDataModuleConfig:
    # height and width should be Union[int, List[int]]
    # but OmegaConf does not support Union of containers
    height: Any = 96
    width: Any = 96
    resolution_milestones: List[int] = field(default_factory=lambda: [])
    default_elevation_deg: float = 0.0
    default_azimuth_deg: float = -180.0
    default_camera_distance: float = 1.2
    default_fovy_deg: float = 60.0
    image_path: str = ""
    use_random_camera: bool = True
    random_camera: dict = field(default_factory=dict)
    rays_noise_scale: float = 2e-3
    batch_size: int = 1
    requires_depth: bool = False
    requires_normal: bool = False
    rays_d_normalize: bool = True
    use_mixed_camera_config: bool = False