Error while generating depth map

[atulshiva]

Downloaded the extension from stable diffusion on my browser however I keep getting numerous errors but some I have fixed, like the vram issue and the depth map not appearing on stable diffusion. Now I got another error while trying to generate a depth map it shows.

Loading weights [e1441589a6] from C:\Users\atuls\Desktop\AI\stable-diffusion-webui\models\Stable-diffusion\v1-5-pruned.ckpt Creating model from config: C:\Users\atuls\Desktop\AI\stable-diffusion-webui\configs\v1-inference.yaml LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. Applying cross attention optimization (Doggettx). Textual inversion embeddings loaded(0): Model loaded in 132.8s (load weights from disk: 88.4s, load config: 0.1s, create model: 2.3s, apply weights to model: 13.9s, apply half(): 2.4s, apply dtype to VAE: 0.1s, load VAE: 0.4s, move model to device: 1.6s, load textual inversion embeddings: 23.4s). Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_beit_large_512.pt Downloading https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt to ./models/midas/dpt_beit_large_512.pt 100%|█████████████████████████████████████████████████████████████████████████████| 1.47G/1.47G [02:06<00:00, 12.5MB/s] initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%| | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 0%| | 0/1 [00:30<?, ?it/s] ERROR: out of memory, could not generate depthmap ! Running 3D Photo Inpainting .. Loading edge model .. Loading depth model .. Loading rgb model .. 0it [00:00, ?it/s] All done. Error completing request Arguments: (0, <PIL.Image.Image image mode=RGB size=2048x2048 at 0x1D9881E8160>, None, '', '', 0, 1, 612, 612, False, False, True, True, True, False, False, 1, False, False, 2.5, 4, 0, False, 0, 1, True, True, 'u2net', False, False, False, 0, 2) {} Traceback (most recent call last): File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, *kwargs)) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(args, **kwargs) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 1146, in run_generate outputs, mesh_fi = run_depthmap(None, outpath, imageArr, imageNameArr, compute_device, model_type, net_width, net_height, match_size, invert_depth, boost, save_depth, show_depth, show_heat, combine_output, combine_output_axis, gen_stereo, gen_anaglyph, stereo_divergence, stereo_fill, stereo_balance, clipdepth, clipthreshold_far, clipthreshold_near, inpaint, inpaint_vids, fnExt, vid_ssaa, background_removal, background_removed_images, save_background_removal_masks, False) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 550, in run_depthmap mesh_fi = run_3dphoto(device, inpaint_imgs, inpaint_depths, inputnames, outpath, fnExt, vid_ssaa, inpaint_vids) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 721, in run_3dphoto return mesh_fi UnboundLocalError: local variable 'mesh_fi' referenced before assignment

Traceback (most recent call last): File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1018, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 935, in postprocess_data if predictions[i] is components._Keywords.FINISHED_ITERATING: IndexError: tuple index out of range

[atulshiva]

Alright so i switched from GPU to CPU and it worked fine but I guess it takes a really long time, however, theres no reason I should not be able to use my graphics card? EDIT: computer crashed before the process was finished i dont even know how far it had progressed but it took 1 hour and 30 minutes to crash.

[graemeniedermayer]

I'm not sure about all the problems, but res101 sometimes has trouble with certain sizes https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/98 . It seems to really like powers of 2 so maybe try 256 by 256?

It looks like you are also doing 3d inpainting. For troubleshooting it might be helpful to see if it works without 3d inpainting on.

[atulshiva]

I'm not sure about all the problems, but res101 sometimes has trouble with certain sizes #98 . It seems to really like powers of 2 so maybe try 256 by 256?

It looks like you are also doing 3d inpainting. For troubleshooting it might be helpful to see if it works without 3d inpainting on.

I did, I tried mostly all the resolution and the low ones but none of the models work expect the last one midasv21(small), but i need higher resolution ones. However i saw another post of someone with an rtx 3090 having the same exact issue it I know that its not an error on my part. How do it fix it?

[atulshiva]

Okay there are two generation buttons, and it looks like i was clicking the "generate video" instead of "generate" but even when i clicked generate it still gave me this error. I got as far as "generating inpainted mesh grab some coffee"

Generating inpainted mesh .. (go make some coffee) .. 0%| | 0/1 [1:07:20<?, ?it/s] Error completing request4%|████████▍ | 1/7 [1:05:37<10:50, 108.34s/it] Arguments: (0, <PIL.Image.Image image mode=RGB size=2048x2048 at 0x24F6F00C460>, None, '', '', 0, 0, 512, 512, False, False, True, True, True, False, False, 1, False, False, 2.5, 4, 0, False, 0, 1, True, True, 'u2net', False, False, False, 0, 2) {} Traceback (most recent call last): File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 550, in run_depthmap mesh_fi = run_3dphoto(device, inpaint_imgs, inpaint_depths, inputnames, outpath, fnExt, vid_ssaa, inpaint_vids) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 693, in run_3dphoto rt_info = write_ply(img, File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions/stable-diffusion-webui-depthmap-script/scripts\inpaint\mesh.py", line 1895, in write_ply info_on_pix, input_mesh, image, depth, edge_ccs = extrapolate(input_mesh, info_on_pix, image, depth, other_edge_with_id, edge_map, edge_ccs, File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions/stable-diffusion-webui-depthmap-script/scripts\inpaint\mesh_tools.py", line 205, in extrapolate depth_edge_output = depth_edge_model.forward_3P(t_mask, t_context, t_rgb, t_disp, t_edge, unit_length=128, File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions/stable-diffusion-webui-depthmap-script/scripts\inpaint\networks.py", line 311, in forward_3P edge_output = self.forward(enlarge_input) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions/stable-diffusion-webui-depthmap-script/scripts\inpaint\networks.py", line 324, in forward x6 = self.decoder_1(torch.cat((x5, x2), dim=1)) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward input = module(input) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1212, in _call_impl result = forward_call(input, **kwargs) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 956, in forward return F.conv_transpose2d( torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 4.00 GiB total capacity; 3.30 GiB already allocated; 0 bytes free; 3.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, *kwargs)) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(args, **kwargs) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 1146, in run_generate outputs, mesh_fi = run_depthmap(None, outpath, imageArr, imageNameArr, compute_device, model_type, net_width, net_height, match_size, invert_depth, boost, save_depth, show_depth, show_heat, combine_output, combine_output_axis, gen_stereo, gen_anaglyph, stereo_divergence, stereo_fill, stereo_balance, clipdepth, clipthreshold_far, clipthreshold_near, inpaint, inpaint_vids, fnExt, vid_ssaa, background_removal, background_removed_images, save_background_removal_masks, False) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 554, in run_depthmap shared.sd_model.cond_stage_model.to(devices.device) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 989, in to return self._apply(convert) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply module._apply(fn) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply module._apply(fn) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply module._apply(fn) [Previous line repeated 5 more times] File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 664, in _apply param_applied = fn(param) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 987, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.51 GiB already allocated; 0 bytes free; 3.52 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Extrapolating border: 14%|████████ | 1/7 [1:06:38<6:39:50, 3998.45s/it] Traceback (most recent call last): File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1018, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "C:\Users\atuls\Desktop\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 935, in postprocess_data if predictions[i] is components._Keywords.FINISHED_ITERATING: IndexError: tuple index out of range

[atulshiva]

Every single time now the errors end of "Extrapolating border: 14%|████████"

[AugmentedRealityCat]

You are asking for more than your GPU can manage in VRAM. 2048x2048 is very large. Try reducing the resolution of the image you are working from, not just the resolution of the depthmap generation - it makes a huge difference in mesh generation as its resolution is based on the RGB colored source texture, and NOT on the depthmap resolution.

Here is the part of your error log that says clearly that this is the source of your problem (now that you use the right button - I made the exact same error at the beginning !). torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 4.00 GiB total capacity; 3.30 GiB already allocated; 0 bytes free; 3.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[atulshiva]

You are asking for more than your GPU can manage in VRAM. 2048x2048 is very large. Try reducing the resolution of the image you are working from, not just the resolution of the depthmap generation - it makes a huge difference in mesh generation as its resolution is based on the RGB colored source texture, and NOT on the depthmap resolution.

Here is the part of your error log that says clearly that this is the source of your problem (now that you use the right button - I made the exact same error at the beginning !). torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 4.00 GiB total capacity; 3.30 GiB already allocated; 0 bytes free; 3.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hey thanks for the response, so what setting do i have to change? Oh so i have to reduce the solution of the resize the image?
Edit: I reduced the size by 50 percent still getting the same error. The only model i can use it midas(small) but i want to use the other models however i keep getting the out of memory error, is there any way to get around this?

[AugmentedRealityCat]

Have you tried scaling down the image first in a separate app, and then load it at this reduced resolution ? Maybe reducing the image size parameters doesn't get to influence the 3d mesh generation code somehow.

Else, if you get the same error after trying all that, it's very simple to fix: use the CPU instead of the GPU.

[atulshiva]

Have you tried scaling down the image first in a separate app, and then load it at this reduced resolution ? Maybe reducing the image size parameters doesn't get to influence the 3d mesh generation code somehow.

Else, if you get the same error after trying all that, it's very simple to fix: use the CPU instead of the GPU.

I resized the image from its original size to 640x640 and then set the internal resolution on the depth in stable difussion to 2048x2048 and it worked for 1 image on res101 model. IDK HOW but it worked for 1 image. However, when i tried to do it with another image it wont work anymore?? Heres it the code for the sucessful one

DepthMap v0.3.8 device: cuda Loading model weights from ./models/leres/res101.pth initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%| | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 896 Adjust factor is: 1.0 Selecting patches ... Target resolution: (1792, 1792, 3) Resulting depthmap resolution will be : (640, 640) patches to process: 7 processing patch 0 / 6 | [ 0 0 560 560] processing patch 1 / 6 | [ 40 160 480 480] processing patch 2 / 6 | [160 160 480 480] processing patch 3 / 6 | [ 0 0 320 320] processing patch 4 / 6 | [ 0 120 320 320] processing patch 5 / 6 | [ 0 240 320 320] processing patch 6 / 6 | [120 0 320 320] 100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.91s/it] Done. Running 3D Photo Inpainting .. Loading edge model .. Loading depth model .. Loading rgb model .. 0%| | 0/1 [00:00<?, ?it/s] Generating inpainted mesh .. (go make some coffee) .. Generating faces: 100%|██████████████████████████████████████████████████████████████████| 7/7 [03:55<00:00, 33.70s/it] Writing mesh file outputs/extras-images\depthmap-0000.ply ...████████████████████████████| 7/7 [03:55<00:00, 26.66s/it] Saving faces: 100%|██████████████████████████████████████████████████████| 5384452/5384452 [00:06<00:00, 808390.90it/s] 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [04:12<00:00, 252.94s/it] All done.

As you can see I dont understand why it wont work anymore? It worked but when I tried it again with the same image this happened, so I know for a fact that something is wrong not with my system but with depth. ALso my computer crashed while using CPU so i hardly think its a good idea to do so. And i have a ryzen 5 5600x

[AugmentedRealityCat]

You should try to discover what are the conditions that makes it work in certain circumstances and not it some others. That would help you find the root cause of the problems you are experiencing. For example, maybe it has to do with VRAM management, and a clue would be that your procedure would always work if you'd run it as the first task just after launching Stable-Diffusion, but would fail after that.

You can also use one of the many online demo for 3d-photo-inpainting then - this works on any machine that can reach the webpage ! Here are 3 of them: https://replicate.com/vt-vl-lab/3d-photo-inpainting https://huggingface.co/spaces/Epoching/3D_Photo_Inpainting https://colab.research.google.com/github/fzantalis/colab_collection/blob/master/3D_Photo_Inpainting.ipynb

By the way, I am just a user like you, I am not a developer for this extension - I am not even a programmer - I am just trying to help you find solutions that would work for you in the current circumstances. If there is indeed a problem with this extension's code, I won't be the person you need to fix it !

[atulshiva]

You should try to discover what are the conditions that makes it work in certain circumstances and not it some others. That would help you find the root cause of the problems you are experiencing. For example, maybe it has to do with VRAM management, and a clue would be that your procedure would always work if you'd run it as the first task just after launching Stable-Diffusion, but would fail after that.

You can also use one of the many online demo for 3d-photo-inpainting then - this works on any machine that can reach the webpage ! Here are 3 of them: https://replicate.com/vt-vl-lab/3d-photo-inpainting https://huggingface.co/spaces/Epoching/3D_Photo_Inpainting https://colab.research.google.com/github/fzantalis/colab_collection/blob/master/3D_Photo_Inpainting.ipynb

By the way, I am just a user like you, I am not a developer for this extension - I am not even a programmer - I am just trying to help you find solutions that would work for you in the current circumstances. If there is indeed a problem with this extension's code, I won't be the person you need to fix it !

Yes thank you man, because of you I've found some good ways to get this extension to work! I changed it to CPU and reduced the resolution which made render faster and did not crash my computer, So i think for now going for GPU is not an option if i want to create depth maps.

[errorsurface]

I'm getting this error if I check Save Depthmap, but it works fine if I don't. Thought it had to do with permissions but I double checked that the output folder (in the case of Batch from directory) is not set to read only.

`DepthMap v0.3.8 device: cuda Loading model weights from ./models/leres/res101.pth initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%| | 0/440 [00:00<?, ?it/s]

wholeImage being processed in : 672 Adjust factor is: 1.9363762102351316 Selecting patches ... Target resolution: (1464, 2602, 3) Resulting depthmap resolution will be : (2160, 3839) patches to process: 4 processing patch 0 / 3 | [ 186 186 1952 1952] processing patch 1 / 3 | [ 682 186 1952 1952] processing patch 2 / 3 | [1399 407 1502 1502] processing patch 3 / 3 | [ 0 496 1322 1322] 0%| | 0/440 [00:05<?, ?it/s] Error completing request Arguments: (2, <PIL.Image.Image image mode=RGB size=3840x2160 at 0x212F646C940>, None, 'C:\Users\sdiff\Documents\outputs\StableDiff\stereo\sd-lesResBoost\Alicia_Footage\frames\source', 'C:\Users\sdiff\Documents\outputs\StableDiff\stereo\sd-lesResBoost\Alicia_Footage\frames\depth', 0, 0, 512, 512, False, False, True, True, True, False, False, 1, True, False, 0.5, 3, 0, False, 0, 1, False, False, 'u2net', False, False, False, 0, 2) {} Traceback (most recent call last): File "C:\stable-diffusion\stable-diffusion-webui-master\venv\lib\site-packages\PIL\JpegImagePlugin.py", line 643, in _save rawmode = RAWMODE[im.mode] KeyError: 'I;16'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\stable-diffusion\stable-diffusion-webui-master\modules\call_queue.py", line 56, in f res = list(func(*args, *kwargs)) File "C:\stable-diffusion\stable-diffusion-webui-master\modules\call_queue.py", line 37, in f res = func(args, **kwargs) File "C:\stable-diffusion\stable-diffusion-webui-master\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 1146, in run_generate outputs, mesh_fi = run_depthmap(None, outpath, imageArr, imageNameArr, compute_device, model_type, net_width, net_height, match_size, invert_depth, boost, save_depth, show_depth, show_heat, combine_output, combine_output_axis, gen_stereo, gen_anaglyph, stereo_divergence, stereo_fill, stereo_balance, clipdepth, clipthreshold_far, clipthreshold_near, inpaint, inpaint_vids, fnExt, vid_ssaa, background_removal, background_removed_images, save_background_removal_masks, False) File "C:\stable-diffusion\stable-diffusion-webui-master\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 447, in run_depthmap images.save_image(Image.fromarray(img_output), path=outpath, basename=basename, seed=None, prompt=None, extension=opts.samples_format, info=info, short_filename=True,no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=None, forced_filename=None) File "C:\stable-diffusion\stable-diffusion-webui-master\modules\images.py", line 581, in save_image _atomically_save_image(image, fullfn_without_extension, ".jpg") File "C:\stable-diffusion\stable-diffusion-webui-master\modules\images.py", line 550, in _atomically_save_image image_to_save.save(temp_file_path, format=image_format, quality=opts.jpeg_quality) File "C:\stable-diffusion\stable-diffusion-webui-master\venv\lib\site-packages\PIL\Image.py", line 2431, in save save_handler(self, fp, filename) File "C:\stable-diffusion\stable-diffusion-webui-master\venv\lib\site-packages\PIL\JpegImagePlugin.py", line 646, in _save raise OSError(msg) from e OSError: cannot write mode I;16 as JPEG

Traceback (most recent call last): File "C:\stable-diffusion\stable-diffusion-webui-master\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "C:\stable-diffusion\stable-diffusion-webui-master\venv\lib\site-packages\gradio\blocks.py", line 1018, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "C:\stable-diffusion\stable-diffusion-webui-master\venv\lib\site-packages\gradio\blocks.py", line 935, in postprocess_data if predictions[i] is components._Keywords.FINISHED_ITERATING: IndexError: tuple index out of range`

[semjon00]

@errorsurface The version v0.3.8 installed on your machine is outdated. Please install the latest version, the problem most likely will resolve.

[Volcanicus]

If using the Depth Tab on version DepthMap v0.4.4 (0d579ddd), no longer an issue; will not work in img2img tab however: there is a tuple index out of range error as described