Closed scooller closed 1 year ago
vladmandic
commented
1 year ago bfloat16 is a new format in torch and its not implemented yet for all functions. i don't even suggest trying it unless you're running torch 2.1 (which is not even released yet).
scooller
commented
1 year ago The strange thing is that it doesn't activate it, I don't know where that came from
vladmandic
commented
1 year ago Settings - compute settings - set fp16.
scooller
commented
1 year ago Thanks but the first error persists, i erase the ven and nothing
22:42:16-508001 INFO Server arguments: [] No module 'xformers'. Proceeding without it. 22:42:22-597778 INFO Libraries loaded 22:42:22-600779 INFO Using data path: C:\Vlad-automatic 22:42:22-609299 INFO Available VAEs: C:\stable-diffusion-webui\models\VAE 13 22:42:22-682418 INFO Available models: C:\stable-diffusion-webui\models\Stable-diffusion 55 2023-06-03 22:42:25,726 - ControlNet - INFO - ControlNet v1.1.215 2023-06-03 22:42:25,855 - ControlNet - INFO - ControlNet v1.1.215 22:42:26-995323 INFO Libraries loaded 22:42:27-248456 INFO Loading UI theme: name=gradio/base style=Auto ╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮ │ C:\Vlad-automatic\launch.py:149 in <module> │ │ │ │ 148 │ │ ❱ 149 instance = start_server(immediate=True, server=None) │ │ 150 while True: │ │ │ │ C:\Vlad-automatic\launch.py:129 in start_server │ │ │ │ 128 else: │ │ ❱ 129 server = server.webui() │ │ 130 if args.profile: │ │ │ │ C:\Vlad-automatic\webui.py:276 in webui │ │ │ │ 275 start_common() │ │ ❱ 276 start_ui() │ │ 277 load_model() │ │ │ │ C:\Vlad-automatic\webui.py:204 in start_ui │ │ │ │ 203 startup_timer.record("scripts before_ui_callback") │ │ ❱ 204 shared.demo = modules.ui.create_ui() │ │ 205 startup_timer.record("ui") │ │ │ │ C:\Vlad-automatic\modules\ui.py:353 in create_ui │ │ │ │ 352 modules.scripts.scripts_current = modules.scripts.scripts_txt2img │ │ ❱ 353 modules.scripts.scripts_txt2img.initialize_scripts(is_img2img=False) │ │ 354 with gr.Blocks(analytics_enabled=False) as txt2img_interface: │ │ │ │ C:\Vlad-automatic\modules\scripts.py:282 in initialize_scripts │ │ │ │ 281 for script_class, path, _basedir, _script_module in auto_processing_scripts + sc │ │ ❱ 282 script = script_class() │ │ 283 script.filename = path │ │ │ │ C:\Vlad-automatic\extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py:26 in __init__ │ │ │ │ 25 self.checkpoint_override = checkpoint_current │ │ ❱ 26 self.generate_button │ │ 27 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'Script' object has no attribute 'generate_button' Press a key to continue . . .
vladmandic
commented
1 year ago That is a completely different issue and we'll known, there are 4 duplicates for it already. Patch is ready, but needs to be merged upstream.
DoughyInTheMiddle
commented
1 year ago Settings - compute settings - set fp16.
I've changed the setting above, yet the System tab still says bf16. Is there somewhere else I need to do this?
All I know is that since I've been using this UI, especially in DB training, I've been using bf16. I would assume it's also being used in the rest of the UI. Was still able to use the UI just fine as of two days ago, but yesterday and today, I can't run even a simple 1024x768 image or it blows up.
Will roll back a couple days to see if that fixes it, but here's my current setup followed by the last error I got.
Version updated: 2023-06-04 hash: 9726b4d2 url: https://github.com/vladmandic/automatic/tree/master
Platform arch: AMD64 cpu: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel system: Windows release: Windows-10-10.0.19044-SP0 python: 3.10.6
Torch 2.0.1+cu118 Autocast half
GPU device: NVIDIA GeForce RTX 3060 (1) (compute_37) (8, 6) cuda: 11.8 cudnn: 8700
Memory optimization none
Cross-attention sdp
Libs xformers: unavailable accelerate: 0.19.0 transformers: 4.29.2
Repos Stable Diffusion: [cf1d67a] 2023-03-25 Taming Transformers: [9e9981c] 2022-11-07 CodeFormer: [7a584fd] 2023-01-18 BLIP: [3a29b74] 2022-09-20 k_diffusion: [c9fe758] 2023-05-21
Device Info active: cuda dtype: torch.bfloat16 vae: torch.bfloat16 unet: torch.float16
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 150/150 [00:27<00:00, 5.41it/s]
15:57:20-043771 ERROR Exception: "upsample_nearest2d_out_frame" not implemented for 'BFloat16'
15:57:20-045755 ERROR Arguments: args=('task(yl95r870ppaekif)', 0, 'a man with a beard and a green shirt is holding his arms up in the air and smiling at the camera with his hands up in the air.', '',
[], <PIL.Image.Image image mode=RGBA size=293x196 at 0x2879965BD30>, None, None, None, None, None, None, 150, 0, 4, 0, 1, False, False, 1, 1, 30, 1.5, 1, 0.01, -1.0, -1.0, 0, 0, 0,
False, 0, 512, 683, 1, 0, 0, 32, 0, '', '', '', [], 0, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 1, 'None', 2, False, 10, 1, 1, 64, False, False, False, False,
False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,
False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,
False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 2048, 128, True, True, True, False, False, 7, 100,
'Constant', 0, 'Constant', 0, 4, False, 'x264', 'blend', 10, 0, 0, False, True, True, True, 'intermediate', 'animation', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit
object at 0x00000285FC200820>, '<ul>\n<li><code>CFG Scale</code> should be 2 or lower.</li>\n</ul>\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p
style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4,
0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use
width and height sliders to set tile size</p>', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, 50, False, 4.0, '', 10.0,
'Linear', 3, False, 30.0, True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True, 'Positive', 0, ', ', 'Generate and always save', 32) kwargs={}
15:57:20-085746 ERROR gradio call: RuntimeError
┌───────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────┐
│ G:\GitHub\Vladmandic\modules\call_queue.py:34 in f │
│ │
│ 33 │ │ │ try: │
│ > 34 │ │ │ │ res = func(*args, **kwargs) │
│ 35 │ │ │ │ progress.record_results(id_task, res) │
│ │
│ G:\GitHub\Vladmandic\modules\img2img.py:174 in img2img │
│ │
│ 173 │ │ if processed is None: │
│ > 174 │ │ │ processed = process_images(p) │
│ 175 │ p.close() │
│ │
│ ... 13 frames hidden ... │
│ │
│ G:\GitHub\Vladmandic\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py:62 in forward │
│ │
│ 61 │ def forward(self, x): │
│ > 62 │ │ x = torch.nn.functional.interpolate(x, scale_factor=2.0, mode="nearest") │
│ 63 │ │ if self.with_conv: │
│ │
│ G:\GitHub\Vladmandic\venv\lib\site-packages\torch\nn\functional.py:3931 in interpolate │
│ │
│ 3930 │ if input.dim() == 4 and mode == "nearest": │
│ > 3931 │ │ return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors) │
│ 3932 │ if input.dim() == 5 and mode == "nearest": │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: "upsample_nearest2d_out_frame" not implemented for 'BFloat16'I rolled back to 561ebc612eef3f802883a6f2091da072405637ca (2023-06-01 20:54) and ran the same image. No issues.
Went into Compute Settings and turned bf16 precision back on and ran it again. Still no issues.
DoughyInTheMiddle
commented
1 year ago Hrm. Changed images, different prompt, ran once no issues. Continued with my typical workflow and then it broke again.
Rolling back again to 4b9924a35abd73c15b822d0008e4d738420159f5 (2023-05-31 12:03).
DoughyInTheMiddle
commented
1 year ago OK, yeah, img2img is broken. Even rolling back to the 5/31 one doesn't work, and I was doing stuff all day long that day. Might go back one more. Completely restarted the server as well and it just stays on bf16 regardless of Compute Settings.
NVidia popped up an update as well I thought maybe........but no.
Very confused why it's not working. Might go back even further just to check.
vladmandic
commented
1 year ago any chance you can delete venv and config.json and start scratch? i'm sure it can be fixed via config, but you'll have to do the leg work for that.
DoughyInTheMiddle
commented
1 year ago I think it is intermittent now. I know for CERTAIN I was closing and fully restarting the server (not just hitting the restart server button), but this last time I rolled back to 9fb93a072828cbc88400d74f5533ec3ad3cdb78f (2023-05-30) it had no issues...but it was still set to fp16. When I switched to bf16, it crashed again. I restarted the server, and it still crashed.
So, I changed it back to fp16 and fully closed and restarted again...no issues. If you wanna close it, go ahead, but if it happens again, I'll ping back.
Issue Description
Hash 549c02a8
Platform
Torch
GPU
Cross-attention
First when it start i have error
13:42:35-850368 ERROR executing callback: C:\Vlad-automatic\extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py app_started_callback: AttributeError ╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮ │ C:\Vlad-automatic\modules\script_callbacks.py:125 in app_started_callbackand after when i use txt2img `13:42:57-557421 ERROR HTTP API: RuntimeError Progress 1.24it/s ━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15% 0:00:03 0:00:02╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮ │ C:\Vlad-automatic\venv\lib\site-packages\anyio\streams\memory.py:98 in receive │ │ │ │ C:\Vlad-automatic\venv\lib\site-packages\anyio\streams\memory.py:93 in receive_nowait │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ WouldBlock
During handling of the above exception, another exception occurred:
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮ │ C:\Vlad-automatic\venv\lib\site-packages\starlette\middleware\base.py:78 in call_next │ │ │ │ C:\Vlad-automatic\venv\lib\site-packages\anyio\streams\memory.py:118 in receive │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ EndOfStream
During handling of the above exception, another exception occurred:
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮ │ C:\Vlad-automatic\modules\middleware.py:72 in exception_handling │ │ │ │ 71 │ │ try: │ │ ❱ 72 │ │ │ return await call_next(req) │ │ 73 │ │ except CancelledError: │ │ │ │ C:\Vlad-automatic\venv\lib\site-packages\starlette\middleware\base.py:84 in call_next │ │ │ │ ... 27 frames hidden ... │ │ │ │ C:\Vlad-automatic\modules\sd_vae_approx.py:24 in forward │ │ │ │ 23 │ │ extra = 11 │ │ ❱ 24 │ │ x = nn.functional.interpolate(x, (x.shape[2] 2, x.shape[3] 2)) │ │ 25 │ │ x = nn.functional.pad(x, (extra, extra, extra, extra)) │ │ │ │ C:\Vlad-automatic\venv\lib\site-packages\torch\nn\functional.py:3931 in interpolate │ │ │ │ 3930 │ if input.dim() == 4 and mode == "nearest": │ │ ❱ 3931 │ │ return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors) │ │ 3932 │ if input.dim() == 5 and mode == "nearest": │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: "upsample_nearest2d_out_frame" not implemented for 'BFloat16' Progress 4.08it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:04 13:43:00-025498 ERROR Exception: "upsample_nearest2d_out_frame" not implemented for 'BFloat16' 13:43:00-026499 ERROR Arguments: args=('task(8dgrcrnbsltbdv4)', 'RAW photo,(high detailed skin:1.2), 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3,MODELSHOOT STYLE,a bust portrait of a girl, beautiful woman, wearing a dress, model photo, contest winner, Bokeh', '((((big hands, un-detailed skin, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime)))), (((ugly mouth, ugly eyes, missing teeth, crooked teeth, close up, cropped, out of frame))), worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck,text,logo,watermark,', [], 20, 12, False, False, 1, 1, 6, 1, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 2048, 128, True, True, True, False, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002D0FABB91B0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002D1576E5DB0>, False, '', 0, False, '', 0.5, True, False, '', 'Lerp', False, False, 0.75, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, 50) kwargs={} 13:43:00-032518 ERROR gradio call: RuntimeError ╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮ │ C:\Vlad-automatic\modules\call_queue.py:34 in f │ │ │ │ 33 │ │ │ try: │ │ ❱ 34 │ │ │ │ res = func(*args, **kwargs) │ │ 35 │ │ │ │ progress.record_results(id_task, res) │ │ │ │ C:\Vlad-automatic\modules\txt2img.py:58 in txt2img │ │ │ │ 57 │ if processed is None: │ │ ❱ 58 │ │ processed = process_images(p) │ │ 59 │ p.close() │ │ │ │ ... 13 frames hidden ... │ │ │ │ C:\Vlad-automatic\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py:62 in forward │ │ │ │ 61 │ def forward(self, x): │ │ ❱ 62 │ │ x = torch.nn.functional.interpolate(x, scale_factor=2.0, mode="nearest") │ │ 63 │ │ if self.with_conv: │ │ │ │ C:\Vlad-automatic\venv\lib\site-packages\torch\nn\functional.py:3931 in interpolate │ │ │ │ 3930 │ if input.dim() == 4 and mode == "nearest": │ │ ❱ 3931 │ │ return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors) │ │ 3932 │ if input.dim() == 5 and mode == "nearest": │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: "upsample_nearest2d_out_frame" not implemented for 'BFloat16'`
Version Platform Description
Windows-10-10.0.22621-SP0