cvlab-kaist / 3DFuse

Official implementation of "Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation"
715 stars 42 forks source link

Problem generating 3D after point cloud #12

Open throttlekitty opened 1 year ago

throttlekitty commented 1 year ago

Running local on Win10 I'm getting the following error when generating 3D after creating the point cloud. This happens at some point during the lora process. I did initially have some problems getting PyTorch3D set up, and I did have to create the conda environment in 3.9, this repo suggests 3.8, in case that's a problem here. I see that ultimately it fails to find this particular image, as it doesn't exist.

...
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Reshaping encoder.mid.attn_1.q.weight for SD format
Reshaping encoder.mid.attn_1.k.weight for SD format
Reshaping encoder.mid.attn_1.v.weight for SD format
Reshaping encoder.mid.attn_1.proj_out.weight for SD format
Reshaping decoder.mid.attn_1.q.weight for SD format
Reshaping decoder.mid.attn_1.k.weight for SD format
Reshaping decoder.mid.attn_1.v.weight for SD format
Reshaping decoder.mid.attn_1.proj_out.weight for SD format
Added Token: ['<0>']
a fuzzy <0>
side view of a fuzzy <0>:   2%|█                                                   | 199/10000 [00:45<37:02,  4.41it/s]
Traceback (most recent call last):
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 929, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\utils.py", line 490, in async_iteration
    return next(iterator)
  File "C:\stable-diffusion\3DFuse\gradio_app.py", line 47, in gen_3d
    yield from model.run_gradio(points,exp_instance_dir)
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 147, in run_gradio
    yield from fuse_3d(**cfgs, poser=poser,model=model,vox=vox,exp_instance_dir=exp_instance_dir, points=points, is_gradio=True)
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 248, in fuse_3d
    vis_routine(metric, y, depth,p,depth_map[0])
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 345, in vis_routine
    metric.put_artifact("img", ".png",prompt, lambda fn: imwrite(fn, im))
  File "C:\stable-diffusion\3DFuse\my\utils\event.py", line 110, in put_artifact
    save_func(fname)
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 345, in <lambda>
    metric.put_artifact("img", ".png",prompt, lambda fn: imwrite(fn, im))
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\functions.py", line 303, in imwrite
    writer = get_writer(uri, format, "i", **kwargs)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\functions.py", line 231, in get_writer
    return format.get_writer(request)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\format.py", line 185, in get_writer
    return self.Writer(self, request)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\format.py", line 221, in __init__
    self._open(**self.request.kwargs.copy())
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\plugins\pillow.py", line 356, in _open
    PillowFormat.Writer._open(self)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\plugins\pillow.py", line 195, in _open
    self._fp = self.request.get_file()
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\request.py", line 331, in get_file
    self._file = open(self.filename, "wb")
OSError: [Errno 22] Invalid argument: 'C:\\stable-diffusion\\3DFuse\\results\\cat_0000\\3d\\img\\step_199_backside_view_of_a_fuzzy_<0>.png'
j0seo commented 1 year ago

Hi, @throttlekitty Thanks for your interest in our work. It seems an error occurs in the part where intermediate results are saved as images in that environment. I cannot currently reproduce the Windows environment, so could you try to erase the lines starting with "metric.put_artifact" in the 'run_3DFuse.py' file and see if it works well?

throttlekitty commented 1 year ago

That ran for a lot longer than before and leads to a similar error with different path formatting now.:

...
Reshaping decoder.mid.attn_1.v.weight for SD format
Reshaping decoder.mid.attn_1.proj_out.weight for SD format
Added Token: ['<0>']
a comfortable <0>
  0%|                                                                                        | 0/10000 [00:00<?, ?it/s]C:\apps\anaconda3\envs\3DFuse\lib\site-packages\torch\nn\functional.py:4227: UserWarning:

Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.

100%|████████████████████████████████████████████████████████████████████████████████| 100/100 [00:13<00:00,  7.67it/s]
Traceback (most recent call last):███████████████████████████████████████████████████| 100/100 [00:13<00:00,  7.68it/s]
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 1111, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 1036, in postprocess_data
    prediction_value = postprocess_update_dict(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 432, in postprocess_update_dict
    prediction_value["value"] = block.postprocess(prediction_value["value"])
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\components.py", line 2152, in postprocess
    y = self.make_temp_copy_if_needed(y)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\components.py", line 245, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\components.py", line 217, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: './results\\bed_0000\\3d/result_10000/video/step_100_.mp4'