search

thygate / stable-diffusion-webui-depthmap-script

High Resolution Depth Maps for Stable Diffusion WebUI
MIT License
1.71k stars 159 forks source link

Error when using dpt_beit. #145

Closed rethink-studios closed 1 year ago

rethink-studios commented 1 year ago

Received this error when:

drag and dropped an image into single image. selected dpt_beit_large_512 (midas3.1) (actually, I get an error when using ANY 3.1 model) Doesn't matter if Match input size is checked or not.. GENERATE

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_beit_large_512.pt Error completing request Arguments: (0, <PIL.Image.Image image mode=RGB size=1024x768 at 0x1DF9D3BA890>, None, '', '', 0, 1, 1024, 768, False, False, True, True, True, False, False, 1, False, False, False, 2.5, 4, 0, False, 0, 1, False, False, 'u2net', False, False, False, 0, 2) {} Traceback (most recent call last): File "D:\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, *kwargs)) File "D:\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(args, kwargs) File "D:\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 1158, in run_generate outputs, mesh_fi = run_depthmap(None, outpath, imageArr, imageNameArr, compute_device, model_type, net_width, net_height, match_size, invert_depth, boost, save_depth, show_depth, show_heat, combine_output, combine_output_axis, gen_stereo, gen_stereotb, gen_anaglyph, stereo_divergence, stereo_fill, stereo_balance, clipdepth, clipthreshold_far, clipthreshold_near, inpaint, inpaint_vids, fnExt, vid_ssaa, background_removal, background_removed_images, save_background_removal_masks, False) File "D:\stable-diffusion-webui\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 226, in run_depthmap model = DPTDepthModel( File "D:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/src\midas\dpt_depth.py", line 102, in init super().init(head, kwargs) File "D:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/src\midas\dpt_depth.py", line 55, in init hooks=hooks[backbone], KeyError: 'beitl16_512'

Traceback (most recent call last): File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1018, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 935, in postprocess_data if predictions[i] is components._Keywords.FINISHED_ITERATING: IndexError: tuple index out of range

thygate commented 1 year ago

seems like the model might be corrupt, try re-downloading dpt_beit_large_512.pt

rethink-studios commented 1 year ago

Nah, just downloaded and replaced dpt_beit and received the same error message..

On Thu, Mar 23, 2023, 6:13 PM Bob Thiry @.***> wrote:

seems like the model might be corrupt, try re-downloading dpt_beit_large_512.pt

— Reply to this email directly, view it on GitHub https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/145#issuecomment-1482030714, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKR6WZSCWARAM4BRH7TVNTW5TKJPANCNFSM6AAAAAAWFXKROA . You are receiving this because you authored the thread.Message ID: <thygate/stable-diffusion-webui-depthmap-script/issues/145/1482030714@ github.com>

thygate commented 1 year ago

deforum_helpers/src\midas\dpt_depth.py

A conflict with deforum perhaps ? It's calling the DPTDepthModel from the deforum midas files ..

15704080 commented 1 year ago

I made the same mistake, DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_beit_large_512.pt Error completing request Arguments: (0, <PIL.Image.Image image mode=RGB size=1024x1024 at 0x1F907E82800>, None, '', '', 0, 1, 512, 512, False, False, True, True, True, False, False, 1, False, False, False, 2.5, 4, 0, False, 0, 1, True, False, 'u2net', False, False, False, 0, 2) {} Traceback (most recent call last): File "F:\SDWEBUI\modules\call_queue.py", line 56, in f res = list(func(*args, *kwargs)) File "F:\SDWEBUI\modules\call_queue.py", line 37, in f res = func(args, kwargs) File "F:\SDWEBUI\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 1158, in run_generate outputs, mesh_fi = run_depthmap(None, outpath, imageArr, imageNameArr, compute_device, model_type, net_width, net_height, match_size, invert_depth, boost, save_depth, show_depth, show_heat, combine_output, combine_output_axis, gen_stereo, gen_stereotb, gen_anaglyph, stereo_divergence, stereo_fill, stereo_balance, clipdepth, clipthreshold_far, clipthreshold_near, inpaint, inpaint_vids, fnExt, vid_ssaa, background_removal, background_removed_images, save_background_removal_masks, False) File "F:\SDWEBUI\extensions\stable-diffusion-webui-depthmap-script\scripts\depthmap.py", line 226, in run_depthmap model = DPTDepthModel( File "F:\SDWEBUI/extensions/deforum/scripts/deforum_helpers/src\midas\dpt_depth.py", line 102, in init super().init(head, kwargs) File "F:\SDWEBUI/extensions/deforum/scripts/deforum_helpers/src\midas\dpt_depth.py", line 55, in init hooks=hooks[backbone], KeyError: 'beitl16_512' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。

Traceback (most recent call last): File "F:\SDWEBUI\py310\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "F:\SDWEBUI\py310\lib\site-packages\gradio\blocks.py", line 1018, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "F:\SDWEBUI\py310\lib\site-packages\gradio\blocks.py", line 935, in postprocess_data if predictions[i] is components._Keywords.FINISHED_ITERATING: IndexError: tuple index out of range

15704080 commented 1 year ago

it may be deforum

thygate commented 1 year ago

Yes it's calling the midas functions inside the deforum midas files instead of the midas files in the depth extension directory, Deforum uses an older version that does not have the beit model yet.

15704080 commented 1 year ago

Can you teach me how to solve it, Thanks a million!

15704080 commented 1 year ago

I used another rented machine and deleted the deforum extension, which seems to be running image What is the graphics card memory required for the 512 model My local 8G will report an error, but renting a 24G is enough to run

rethink-studios commented 1 year ago
  1. I need Deforum to work
  2. Does this mean with the latest Deforum update, the bug is fixed?
  3. Does this mean my Deforum dpt_beit needs downloading again?

Please advise...

On Sun, Mar 26, 2023, 1:31 AM 15704080 @.***> wrote:

I used another rented machine and deleted the deform extension, which seems to be running [image: image] https://user-images.githubusercontent.com/128477008/227759441-9cffd76e-2d2e-4c36-9899-0f9bb11a6b57.png

— Reply to this email directly, view it on GitHub https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/145#issuecomment-1484008939, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKR6WYTH2VMEIPNHGQMZXTW57PETANCNFSM6AAAAAAWFXKROA . You are receiving this because you authored the thread.Message ID: <thygate/stable-diffusion-webui-depthmap-script/issues/145/1484008939@ github.com>

fabskills commented 1 year ago

Hi have same issue, how to resolve this ?

thygate commented 1 year ago

Possible quick fix :

Rename the folder extensions/stable-diffusion-webui-depthmap-script/scripts/midas/ to extensions/stable-diffusion-webui-depthmap-script/scripts/dmidas/

and edit lines 40 to 43 in depthmap.py to change the midas imports to start with "dmidas" too.

change from

# midas imports
from midas.dpt_depth import DPTDepthModel
from midas.midas_net import MidasNet
from midas.midas_net_custom import MidasNet_small
from midas.transforms import Resize, NormalizeImage, PrepareForNet

to

# midas imports
from dmidas.dpt_depth import DPTDepthModel
from dmidas.midas_net import MidasNet
from dmidas.midas_net_custom import MidasNet_small
from dmidas.transforms import Resize, NormalizeImage, PrepareForNet

Hopefully this avoids the conflict with deforum, If someone can verify I will include the change in the next version.

rethink-studios commented 1 year ago

I edited the file, and here are the results:

dpt_beit_large_512:

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_beit_large_512.pt https://mailtrack.io/trace/link/1a4b5c1f7e52e65947aa1c1a325a4cf8387c852b?url=http%3A%2F%2Fdpt_beit_large_512.pt&userId=656862&signature=9b144c0227260dcb initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%| | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 Adjust factor is: 1.0 Selecting patches ... Target resolution: (2304, 3072, 3) Resulting depthmap resolution will be : (768, 1024) patches to process: 22

dpt_beit_large_384:

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_beit_large_384.pt https://mailtrack.io/trace/link/8b510d40798a91bda28d891f07f8049fd44d36be?url=http%3A%2F%2Fdpt_beit_large_384.pt&userId=656862&signature=22e00779f5e206e2 initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%| | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 Adjust factor is: 1.0221465076660987 Selecting patches ... Target resolution: (2355, 3140, 3) Resulting depthmap resolution will be : (768, 1024)

dpt_large-midas-2f21e586.pt https://mailtrack.io/trace/link/c7a66b1ccf7fe51dfb6027bf905ea3d610b4c0c3?url=http%3A%2F%2Fdpt_large-midas-2f21e586.pt&userId=656862&signature=899040c0c03fc0e2

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_large-midas-2f21e586.pt https://mailtrack.io/trace/link/c4bbbe3eb40a189e6986edc85aaff8d688ea3fa5?url=http%3A%2F%2Fdpt_large-midas-2f21e586.pt&userId=656862&signature=91f37a23c20f35d3 Downloading https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt https://mailtrack.io/trace/link/873ead3c2bb24425ceed30c2e2bc362c68a4776b?url=https%3A%2F%2Fgithub.com%2Fintel-isl%2FDPT%2Freleases%2Fdownload%2F1_0%2Fdpt_large-midas-2f21e586.pt&userId=656862&signature=59025966fb63c628 to ./models/midas/dpt_large-midas-2f21e586.pt https://mailtrack.io/trace/link/fd9b25ab335fa42b631f41a4a46fcd12942e8136?url=http%3A%2F%2Fdpt_large-midas-2f21e586.pt&userId=656862&signature=4259ad6b7ce1746a 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.28G/1.28G [01:56<00:00, 11.8MB/s] initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%|

                              | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 Adjust factor is: 1.0221465076660987 Selecting patches ... Target resolution: (2355, 3140, 3) Resulting depthmap resolution will be : (768, 1024)

dpt_hybrid-midas-501f0c75:

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/dpt_hybrid-midas-501f0c75.pt https://mailtrack.io/trace/link/f0e8ecc8c7755a63a8684735e8be7aa185d2e77c?url=http%3A%2F%2Fdpt_hybrid-midas-501f0c75.pt&userId=656862&signature=3e4299ee8ad61929 Downloading https://github.com/intel-isl/DPT/releases/download/1_0/dpt_hybrid-midas-501f0c75.pt https://mailtrack.io/trace/link/531d65d309503dedf2bfe1861d26b28d23635683?url=https%3A%2F%2Fgithub.com%2Fintel-isl%2FDPT%2Freleases%2Fdownload%2F1_0%2Fdpt_hybrid-midas-501f0c75.pt&userId=656862&signature=6687862775c946a4 to ./models/midas/dpt_hybrid-midas-501f0c75.pt https://mailtrack.io/trace/link/b86334012219bf55b1d535896fecd37317c5ea38?url=http%3A%2F%2Fdpt_hybrid-midas-501f0c75.pt&userId=656862&signature=2c14b75eaed8d883 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 470M/470M [00:41<00:00, 11.8MB/s] initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%|

                              | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 Adjust factor is: 1.0221465076660987 Selecting patches ... Target resolution: (2355, 3140, 3) Resulting depthmap resolution will be : (768, 1024)

midas_v21-f6b98070.pt https://mailtrack.io/trace/link/bc477c94aa38fb286fc0da08af9cc5489d4b8274?url=http%3A%2F%2Fmidas_v21-f6b98070.pt&userId=656862&signature=f04a732e2aaadcf3

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/midas_v21-f6b98070.pt https://mailtrack.io/trace/link/24ff6dd97a67c2d7edc12407c00404a8913de976?url=http%3A%2F%2Fmidas_v21-f6b98070.pt&userId=656862&signature=fbbd64eaa520758a Loading weights: ./models/midas/midas_v21-f6b98070.pt https://mailtrack.io/trace/link/5cac3b45f74d4a66f105cb7f2fd7390b2102deee?url=http%3A%2F%2Fmidas_v21-f6b98070.pt&userId=656862&signature=b893118b006cbff2 Using cache found in C:\Users\RETHINK/.cache\torch\hub\facebookresearch_WSL-Images_main initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%|

                              | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 Adjust factor is: 1.0221465076660987 Selecting patches ... Target resolution: (2355, 3140, 3) Resulting depthmap resolution will be : (768, 1024)

midas_v21_small-70d6b9c8.pt

DepthMap v0.3.8 device: cuda Loading model weights from ./models/midas/midas_v21_small-70d6b9c8.pt Loading weights: ./models/midas/midas_v21_small-70d6b9c8.pt Using cache found in C:\Users\RETHINK/.cache\torch\hub\rwightman_gen-efficientnet-pytorch_master initialize network with normal loading the model from ./models/pix2pix\latest_net_G.pth Computing depthmap(s) .. 0%|

                              | 0/1 [00:00<?, ?it/s]

wholeImage being processed in : 1536 Adjust factor is: 1.0221465076660987 Selecting patches ... Target resolution: (2355, 3140, 3) Resulting depthmap resolution will be : (768, 1024)

On Thu, Mar 30, 2023 at 12:13 PM Bob Thiry @.***> wrote:

Possible quick fix :

Rename the folder extensions/stable-diffusion-webui-depthmap-script/scripts/midas/ to extensions/stable-diffusion-webui-depthmap-script/scripts/dmidas/

and edit lines 40 to 43 in depthmap.py to change the midas imports to start with "dmidas" too.

change from

midas imports

from midas.dpt_depth import DPTDepthModel from midas.midas_net import MidasNet from midas.midas_net_custom import MidasNet_small from midas.transforms import Resize, NormalizeImage, PrepareForNet

to

midas imports

from dmidas.dpt_depth import DPTDepthModel from dmidas.midas_net import MidasNet from dmidas.midas_net_custom import MidasNet_small from dmidas.transforms import Resize, NormalizeImage, PrepareForNet

Hopefully this avoids the conflict with deforum, If someone can verify I will include the change in the next version.

— Reply to this email directly, view it on GitHub https://mailtrack.io/trace/link/4d0756b4b1839d8312767cd3dfeed40ad17801ef?url=https%3A%2F%2Fgithub.com%2Fthygate%2Fstable-diffusion-webui-depthmap-script%2Fissues%2F145%23issuecomment-1490651272&userId=656862&signature=b8695b7323af7313, or unsubscribe https://mailtrack.io/trace/link/e1e38bbfe26237dcea664cb135655fcd26da1c46?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOKR6W5I2KIG76GZI6FPEUDW6W5K5ANCNFSM6AAAAAAWFXKROA&userId=656862&signature=d1d9a9d01b2e0c60 . You are receiving this because you authored the thread.Message ID: <thygate/stable-diffusion-webui-depthmap-script/issues/145/1490651272@ github.com>

-- Brian Bullock

Founder / Director of CGI

RETHINK Studios

https://www.rethinkstudios.tv/ https://mailtrack.io/trace/link/be054f466cb89c9a5bce894e49ccc5ea0b51a101?url=https%3A%2F%2Fwww.rethinkstudios.tv%2F&userId=656862&signature=c9c8d9320883826e 708-289-1059 @.*** https://ai.rethinkstudios.tv/ https://mailtrack.io/trace/link/59811be2f746323ad85543c71e51f74ef14445c1?url=https%3A%2F%2Fai.rethinkstudios.tv%2F&userId=656862&signature=90858fe4763096fc [image: linkedin] https://mailtrack.io/trace/link/20d7cec18f6f4421d715d958eca5be9992d60382?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fbrian-bullock-9aa0512%2F&userId=656862&signature=fa4d52145a050da9 [image: instagram] https://mailtrack.io/trace/link/ed0d4f4ddc4fee44356f9e506434aeb9f7a650cf?url=https%3A%2F%2Fwww.instagram.com%2Fbrian.bullock.796%2F&userId=656862&signature=71ffb67ee309136c

thygate commented 1 year ago

Looks like you tried every midas model and were using boost .. So it's working now then ?

boost can take a while, are you killing it ?

rethink-studios commented 1 year ago

So far so good. Will test dpt_beit_large_512.pt on an image sequence and see where it goes.. Hang tight..

On Fri, Mar 31, 2023 at 5:21 PM Bob Thiry @.***> wrote:

So it's working now then ?

— Reply to this email directly, view it on GitHub https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/145#issuecomment-1492674599, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKR6W42HDEZWAKVYKWCHTLW65KHFANCNFSM6AAAAAAWFXKROA . You are receiving this because you authored the thread.Message ID: <thygate/stable-diffusion-webui-depthmap-script/issues/145/1492674599@ github.com>

-- Brian Bullock

Founder / Director of CGI

RETHINK Studios

https://www.rethinkstudios.tv/ https://mailtrack.io/trace/link/c49d69f99e7a20a68b6b1b8145d6cebdc3113b34?url=https%3A%2F%2Fwww.rethinkstudios.tv%2F&userId=656862&signature=b6ced1f686a8476f 708-289-1059 @.*** https://ai.rethinkstudios.tv/ https://mailtrack.io/trace/link/f37b7805cd6bc7bc6c531a55114caef04b3a9e24?url=https%3A%2F%2Fai.rethinkstudios.tv%2F&userId=656862&signature=0eaab52824ff35df [image: linkedin] https://mailtrack.io/trace/link/6cf793bdbe05a1b6661a97eeee2566ea351be43c?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fbrian-bullock-9aa0512%2F&userId=656862&signature=54f584e65152914d [image: instagram] https://mailtrack.io/trace/link/b4457a7d7f78bdb7dd42d5e7192f405b62c9aea6?url=https%3A%2F%2Fwww.instagram.com%2Fbrian.bullock.796%2F&userId=656862&signature=3ce23f3672138ebb

rethink-studios commented 1 year ago

imgSeq is progressing. Great work!

On Fri, Mar 31, 2023 at 5:36 PM Brian Bullock @.***> wrote:

So far so good. Will test dpt_beit_large_512.pt on an image sequence and see where it goes.. Hang tight..

On Fri, Mar 31, 2023 at 5:21 PM Bob Thiry @.***> wrote:

So it's working now then ?

— Reply to this email directly, view it on GitHub https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/145#issuecomment-1492674599, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKR6W42HDEZWAKVYKWCHTLW65KHFANCNFSM6AAAAAAWFXKROA . You are receiving this because you authored the thread.Message ID: <thygate/stable-diffusion-webui-depthmap-script/issues/145/1492674599@ github.com>

-- Brian Bullock

Founder / Director of CGI

RETHINK Studios

https://www.rethinkstudios.tv/ https://mailtrack.io/trace/link/c49d69f99e7a20a68b6b1b8145d6cebdc3113b34?url=https%3A%2F%2Fwww.rethinkstudios.tv%2F&userId=656862&signature=b6ced1f686a8476f 708-289-1059 @.*** https://ai.rethinkstudios.tv/ https://mailtrack.io/trace/link/f37b7805cd6bc7bc6c531a55114caef04b3a9e24?url=https%3A%2F%2Fai.rethinkstudios.tv%2F&userId=656862&signature=0eaab52824ff35df [image: linkedin] https://mailtrack.io/trace/link/6cf793bdbe05a1b6661a97eeee2566ea351be43c?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fbrian-bullock-9aa0512%2F&userId=656862&signature=54f584e65152914d [image: instagram] https://mailtrack.io/trace/link/b4457a7d7f78bdb7dd42d5e7192f405b62c9aea6?url=https%3A%2F%2Fwww.instagram.com%2Fbrian.bullock.796%2F&userId=656862&signature=3ce23f3672138ebb

-- Brian Bullock

Founder / Director of CGI

RETHINK Studios

https://www.rethinkstudios.tv/ 708-289-1059 @.*** https://ai.rethinkstudios.tv/ [image: linkedin] https://www.linkedin.com/in/brian-bullock-9aa0512/ [image: instagram] https://www.instagram.com/brian.bullock.796/

enn-nafnlaus commented 1 year ago

I get this issue too, and deleting Deforum did the trick.

To any devs: is this a Deforum bug, a Depthmap bug, or something mutual? Just wondering which side should have a task for fixing it.

thygate commented 1 year ago

@enn-nafnlaus Not really a bug, more of a collision.

Both extensions use midas and include a copy of the source with the extension. Both use the same name for the folder 'midas', added to the pythonpath, only deforum uses an older version that does not yet include the beit model. Python will use the first copy it finds.

Renaming the folder and the imports as described above will fix the name collision. I will incorporate the changes in the next version of the extension.

enn-nafnlaus commented 1 year ago

Leaving a problem around that's going to trip people up unless they happen to visit the github page and find this open issue is not an acceptable solution. This needs to be fixed in software, either on the depthmap side or the deforum side.

enn-nafnlaus commented 1 year ago

On a side note, is anyone actually getting anything decent out of the Midas 3.1 512 model? All the results I've gotten thusfar have been awful, just sheer monstrosities. I'm trying the Midas 3.1 384 model right now to see if it's any better.

ED: Nope, 384 is awful too.

thygate commented 1 year ago

fixed