Memory size requirement

python launch.py --config custom/threestudio-gaussiandreamer/configs/gaussiandreamer.yaml --train --gpu 0 system.prompt_processor.prompt="a tank" system.geometry.geometry_convert_from="shap-e:a tank"
Ignoring custom/.ipynb_checkpoints as it is inside .ipynb_checkpoints.

Import times for custom modules:
   0.0 seconds (IMPORT FAILED): custom/.ipynb_checkpoints
   0.0 seconds: custom/shap-e
   0.0 seconds: custom/threestudio-mvdream
   0.1 seconds: custom/threestudio-shap-e
   1.3 seconds: custom/threestudio-gaussiandreamer
  34.7 seconds: custom/threestudio-lrm

Global seed set to 0
[INFO] Loading shap-e guidance ...
[INFO] Loaded shap-e guidance!
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:06<00:00, 10.65it/s]
/ai/data/threestudio/custom/shap-e/shap_e/models/stf/renderer.py:286: UserWarning: exception rendering with PyTorch3D: No module named 'pytorch3d'
  warnings.warn(f"exception rendering with PyTorch3D: {exc}")
/ai/data/threestudio/custom/shap-e/shap_e/models/stf/renderer.py:287: UserWarning: falling back on native PyTorch renderer, which does not support full gradients
  warnings.warn(
[INFO] Transforming point cloud with directions: -y,+x,+z
[INFO] Number of points at initialisation:10534
[INFO] [Note] Gaussian Splatting doesn't support material and background now.
[INFO] Loading Stable Diffusion ...
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:49<00:00, 12.38s/it]
[INFO] Loaded Stable Diffusion!
[INFO] Using prompt [a tank] and negative prompt [ugly, bad anatomy, blurry, pixelated obscure, unnatural colors, poor lighting, dull, and unclear, cropped, lowres, low quality, artifacts, duplicate, morbid, mutilated, poorly drawn face, deformed, dehydrated, bad proportions, unfocused]
[INFO] Using view-dependent prompts [side]:[a tank, side view] [front]:[a tank, front view] [back]:[a tank, back view] [overhead]:[a tank, overhead view]
[INFO] ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[INFO] 
  | Name       | Type                           | Params
--------------------------------------------------------------
0 | geometry   | GaussianBaseModel              | 147 K 
1 | material   | NoMaterial                     | 0     
2 | background | NeuralEnvironmentMapBackground | 448   
3 | renderer   | DiffGaussian                   | 0     
--------------------------------------------------------------
147 K     Trainable params
0         Non-trainable params
147 K     Total params
0.592     Total estimated model params size (MB)
[INFO] Validation results will be saved to outputs/gs-sds-generation/a_tank@20240820-004500/save
/root/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py:105: UserWarning: Total length of `CombinedLoader` across ranks is zero. Please make sure this was your intention.
  rank_zero_warn(
Epoch 0: : 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/root/miniconda3/envs/threestudio/lib/python3.9/multiprocessing/resource_sharer.py", line 138, in _serve
    with self._listener.accept() as conn:
  File "/root/miniconda3/envs/threestudio/lib/python3.9/multiprocessing/connection.py", line 466, in accept
    answer_challenge(c, self._authkey)
  File "/root/miniconda3/envs/threestudio/lib/python3.9/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/root/miniconda3/envs/threestudio/lib/python3.9/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/root/miniconda3/envs/threestudio/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/threestudio/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Killed
Is 100GB of memory not enough? The process is still killed.
cxh0519 / threestudio-gaussiandreamer

Memory size requirement #12
