ashawkey / stable-dreamfusion

Text-to-3D & Image-to-3D & Mesh Exportation with NeRF + Diffusion.
Apache License 2.0
8.32k stars 734 forks source link

Issues when running in Google Colab #76

Closed frzylucas closed 2 years ago

frzylucas commented 2 years ago

I'm trying to run some experiments with Stable DreamFusion but I am struggling to get it to work. All seems to be going well until I get to the training section. That's where I keep running into issues. When running this part I get the following output:

Namespace(H=800, O=True, O2=False, W=800, albedo_iters=1000, angle_front=60, angle_overhead=30, backbone='grid', bg_radius=1.4, bound=1, ckpt='scratch', cuda_ray=True, density_thresh=10, dir_text=True, dt_gamma=0, eval_interval=10, fovy=60, fovy_range=[40, 70], fp16=True, gui=False, guidance='stable-diffusion', h=64, iters=5000, jitter_pose=False, lambda_entropy=0.0001, lambda_opacity=0, lambda_orient=0.01, lambda_smooth=0, light_phi=0, light_theta=60, lr=0.001, max_ray_batch=4096, max_spp=1, max_steps=1024, min_near=0.1, negative='', num_steps=64, radius=3, radius_range=[1.0, 1.5], save_mesh=True, seed=0, suppress_face=False, test=False, text='a DSLR photo of a delicious hamburger', uniform_sphere_rate=0.5, update_extra_interval=16, upsample_steps=64, w=64, workspace='trial')
NeRFNetwork(
  (encoder): GridEncoder: input_dim=3 num_levels=16 level_dim=2 resolution=16 -> 2048 per_level_scale=1.3819 params=(903480, 2) gridtype=tiled align_corners=False
  (sigma_net): MLP(
    (net): ModuleList(
      (0): Linear(in_features=32, out_features=64, bias=True)
      (1): Linear(in_features=64, out_features=64, bias=True)
      (2): Linear(in_features=64, out_features=4, bias=True)
    )
  )
  (encoder_bg): FreqEncoder: input_dim=3 degree=6 output_dim=39
  (bg_net): MLP(
    (net): ModuleList(
      (0): Linear(in_features=39, out_features=64, bias=True)
      (1): Linear(in_features=64, out_features=3, bias=True)
    )
  )
)
Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with: 

pip install accelerate

.
[INFO] try to load hugging face access token from the default place, make sure you have run `huggingface-cli login`.
[INFO] loading stable diffusion...
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_errors.py](https://localhost:8080/#) in hf_raise_for_status(response, endpoint_name)
    212     try:
--> 213         response.raise_for_status()
    214     except HTTPError as e:

7 frames
HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/vae/diffusion_pytorch_model.bin

The above exception was the direct cause of the following exception:

HfHubHTTPError                            Traceback (most recent call last)
HfHubHTTPError: <class 'requests.exceptions.HTTPError'> (Request ID: Pb97D2MjCZ1M8mc-EwiMB)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/diffusers/modeling_utils.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    425             except HTTPError as err:
    426                 raise EnvironmentError(
--> 427                     "There was a specific connection error when trying to load"
    428                     f" {pretrained_model_name_or_path}:\n{err}"
    429                 )

OSError: There was a specific connection error when trying to load runwayml/stable-diffusion-v1-5:
<class 'requests.exceptions.HTTPError'> (Request ID: Pb97D2MjCZ1M8mc-EwiMB)

I've tried accounting for the missing accelerate library by adding ! pip install accelerate to the dependencies step, but that did not help. On an earlier attempt, this seemed to be the only error I was dealing with, but now there is obviously more to it. Granted, I don't really know what I'm doing, so excuse me if the answer to my question is obvious. Thanks in advance.

ashawkey commented 2 years ago

@frzylucas Hi, have you entered your huggingface access token in the third code block? image

frzylucas commented 2 years ago

@ashawkey Thank you for the reply! I do remember running that block and successfully logging in when I got that output. I tried running the whole thing again just now just to make sure and am getting a different result, not sure what is causing it because, to my knowledge, I did everything the same way. I'll attach it anyway, maybe it's helpful.

 Namespace(H=800, O=True, O2=False, W=800, albedo_iters=1000, angle_front=60, angle_overhead=30, backbone='grid', bg_radius=1.4, bound=1, ckpt='scratch', cuda_ray=True, density_thresh=10, dir_text=True, dt_gamma=0, eval_interval=10, fovy=60, fovy_range=[40, 70], fp16=True, gui=False, guidance='stable-diffusion', h=64, iters=5000, jitter_pose=False, lambda_entropy=0.0001, lambda_opacity=0, lambda_orient=0.01, lambda_smooth=0, light_phi=0, light_theta=60, lr=0.001, max_ray_batch=4096, max_spp=1, max_steps=1024, min_near=0.1, negative='', num_steps=64, radius=3, radius_range=[1.0, 1.5], save_mesh=True, seed=0, suppress_face=False, test=False, text='a DSLR photo of a delicious hamburger', uniform_sphere_rate=0.5, update_extra_interval=16, upsample_steps=64, w=64, workspace='trial')
NeRFNetwork(
  (encoder): GridEncoder: input_dim=3 num_levels=16 level_dim=2 resolution=16 -> 2048 per_level_scale=1.3819 params=(903480, 2) gridtype=tiled align_corners=False
  (sigma_net): MLP(
    (net): ModuleList(
      (0): Linear(in_features=32, out_features=64, bias=True)
      (1): Linear(in_features=64, out_features=64, bias=True)
      (2): Linear(in_features=64, out_features=4, bias=True)
    )
  )
  (encoder_bg): FreqEncoder: input_dim=3 degree=6 output_dim=39
  (bg_net): MLP(
    (net): ModuleList(
      (0): Linear(in_features=39, out_features=64, bias=True)
      (1): Linear(in_features=64, out_features=3, bias=True)
    )
  )
)
[INFO] try to load hugging face access token from the default place, make sure you have run `huggingface-cli login`.
[INFO] loading stable diffusion...
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_errors.py:213 in                   │
│ hf_raise_for_status                                                                              │
│                                                                                                  │
│   210 │   </Tip>                                                                                 │
│   211 │   """                                                                                    │
│   212 │   try:                                                                                   │
│ ❱ 213 │   │   response.raise_for_status()                                                        │
│   214 │   except HTTPError as e:                                                                 │
│   215 │   │   error_code = response.headers.get("X-Error-Code")                                  │
│   216                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.7/dist-packages/requests/models.py:941 in raise_for_status                │
│                                                                                                  │
│   938 │   │   │   http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, rea   │
│   939 │   │                                                                                      │
│   940 │   │   if http_error_msg:                                                                 │
│ ❱ 941 │   │   │   raise HTTPError(http_error_msg, response=self)                                 │
│   942 │                                                                                          │
│   943 │   def close(self):                                                                       │
│   944 │   │   """Releases the connection back to the pool. Once this method has been             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
HTTPError: 403 Client Error: Forbidden for url: 
https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/vae/diffusion_pytorch_model.bin

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.7/dist-packages/diffusers/modeling_utils.py:405 in from_pretrained        │
│                                                                                                  │
│   402 │   │   │   │   │   use_auth_token=use_auth_token,                                         │
│   403 │   │   │   │   │   user_agent=user_agent,                                                 │
│   404 │   │   │   │   │   subfolder=subfolder,                                                   │
│ ❱ 405 │   │   │   │   │   revision=revision,                                                     │
│   406 │   │   │   │   )                                                                          │
│   407 │   │   │                                                                                  │
│   408 │   │   │   except RepositoryNotFoundError:                                                │
│                                                                                                  │
│ /usr/local/lib/python3.7/dist-packages/huggingface_hub/file_download.py:1057 in hf_hub_download  │
│                                                                                                  │
│   1054 │   │   │   │   │   url=url,                                                              │
│   1055 │   │   │   │   │   use_auth_token=use_auth_token,                                        │
│   1056 │   │   │   │   │   proxies=proxies,                                                      │
│ ❱ 1057 │   │   │   │   │   timeout=etag_timeout,                                                 │
│   1058 │   │   │   │   )                                                                         │
│   1059 │   │   │   except EntryNotFoundError as http_error:                                      │
│   1060 │   │   │   │   # Cache the non-existence of the file and raise                           │
│                                                                                                  │
│ /usr/local/lib/python3.7/dist-packages/huggingface_hub/file_download.py:1359 in                  │
│ get_hf_file_metadata                                                                             │
│                                                                                                  │
│   1356 │   │   proxies=proxies,                                                                  │
│   1357 │   │   timeout=timeout,                                                                  │
│   1358 │   )                                                                                     │
│ ❱ 1359 │   hf_raise_for_status(r)                                                                │
│   1360 │                                                                                         │
│   1361 │   # Return                                                                              │
│   1362 │   return HfFileMetadata(                                                                │
│                                                                                                  │
│ /usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_errors.py:254 in                   │
│ hf_raise_for_status                                                                              │
│                                                                                                  │
│   251 │   │                                                                                      │
│   252 │   │   # Convert `HTTPError` into a `HfHubHTTPError` to display request information       │
│   253 │   │   # as well (request id and/or server error message)                                 │
│ ❱ 254 │   │   raise HfHubHTTPError(str(HTTPError), response=response) from e                     │
│   255                                                                                            │
│   256                                                                                            │
│   257 def _raise_for_status(response):                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
HfHubHTTPError: <class 'requests.exceptions.HTTPError'> (Request ID: BoMm2nK4Zsy0W5sA5OFjn)

During handling of the above exception, another exception occurred:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /content/stable-dreamfusion/stable-dreamfusion/main.py:139 in <module>                           │
│                                                                                                  │
│   136 │   │                                                                                      │
│   137 │   │   if opt.guidance == 'stable-diffusion':                                             │
│   138 │   │   │   from nerf.sd import StableDiffusion                                            │
│ ❱ 139 │   │   │   guidance = StableDiffusion(device)                                             │
│   140 │   │   elif opt.guidance == 'clip':                                                       │
│   141 │   │   │   from nerf.clip import CLIP                                                     │
│   142 │   │   │   guidance = CLIP(device)                                                        │
│                                                                                                  │
│ /content/stable-dreamfusion/nerf/sd.py:39 in __init__                                            │
│                                                                                                  │
│    36 │   │   print(f'[INFO] loading stable diffusion...')                                       │
│    37 │   │                                                                                      │
│    38 │   │   # 1. Load the autoencoder model which will be used to decode the latents into im   │
│ ❱  39 │   │   self.vae = AutoencoderKL.from_pretrained("runwayml/stable-diffusion-v1-5", subfo   │
│    40 │   │                                                                                      │
│    41 │   │   # 2. Load the tokenizer and text encoder to tokenize and encode the text.          │
│    42 │   │   self.tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")    │
│                                                                                                  │
│ /usr/local/lib/python3.7/dist-packages/diffusers/modeling_utils.py:427 in from_pretrained        │
│                                                                                                  │
│   424 │   │   │   │   )                                                                          │
│   425 │   │   │   except HTTPError as err:                                                       │
│   426 │   │   │   │   raise EnvironmentError(                                                    │
│ ❱ 427 │   │   │   │   │   "There was a specific connection error when trying to load"            │
│   428 │   │   │   │   │   f" {pretrained_model_name_or_path}:\n{err}"                            │
│   429 │   │   │   │   )                                                                          │
│   430 │   │   │   except ValueError:                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OSError: There was a specific connection error when trying to load runwayml/stable-diffusion-v1-5:
<class 'requests.exceptions.HTTPError'> (Request ID: BoMm2nK4Zsy0W5sA5OFjn)
ashawkey commented 2 years ago

@frzylucas I guess this is the reason: https://news.ycombinator.com/item?id=32556641 You should instead go to check v1.5 though.

frzylucas commented 2 years ago

@ashawkey Yup, was able to run the model smoothly after doing this. Thank you so much!