AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
140.88k stars 26.65k forks source link

[Bug]: "`Error: command buffer exited with error status.` Error and Other While Simultaneously Generating an Image at the 512x512 Resolution and Upscaling It to the 2048x2048 Resolution #9973

Open id88viper88id opened 1 year ago

id88viper88id commented 1 year ago

Is there an existing issue for this?

What happened?

In the process of simultaneously generating an image at the 512x512 resolution and upscaling it to the 2048x2048 resolution by using „Hires. fix”, the image generation process encountered two types of errors. While the image generation process was ongoing with no signs of any issues in the web browser interface of Stable Diffusion 22bcc7b, the first one of those errors, which occurred and was printed out in the Terminal numerous times, was Error: command buffer exited with error status. This particular error was preceded by the initial image generation progress bar, which did not advance from 0% at all, being at the undefined (i.e. ?) value stage of iterations per second. It if directly after that initial image generation progress that the first Error: command buffer exited with error status. error appeared. What followed that error in the Terminal directly is as follows:

The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d2ab5500>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1

All those errors (always starting with Error: command buffer exited with error status. again and again) occur repeatedly and numerous times. As the image generation in the web browser interface progresses undisturbed in any way, over time various instances of those errors, reoccurring in the Terminal, are preceded by single image generation progress bars at various stages of the image generation. What I'm talking about looked like this in particular, following the first instance of the Error: command buffer exited with error status. error:

3%|█▎                                       | 1/30 [03:00<1:27:17, 180.61s/it]Error: command buffer exited with error status. | 31/60 [07:13<26:24, 54.63s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc21a10>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
  7%|██▋                                      | 2/30 [05:59<1:23:53, 179.78s/it]Error: command buffer exited with error status. | 32/60 [10:12<42:56, 92.00s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2adfa2bb0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1

, and so on, with some of the image generation progress bars being at more advanced stages of the image generation process (the entire Terminal console print out, from the very beginning, can be found in the „Console logs” section). What's interesting (and at the same time I do not know what it means) is that the AGXG14XFamilyCommandBuffer values are different from each other, presumably at each individual instance of the Error: command buffer exited with error status. error. Eventually, after numerous reoccurrences of those errors, the final error at the very end is: RuntimeError: Invalid buffer size: 16.00 GB. As the entire image generation and upscaling process concluded in the Terminal with the final RuntimeError: Invalid buffer size: 16.00 GB error as said above, in the web browser interface it concluded so that while in the process of the image generation process there was an emerging image in the window for the image, at the end the said window was empty, and the link under that window to download the image that was being generated was not there. When I clicked the „Save” icon located under the said window, no image was saved to the default download location, and the following error was printed out in the Terminal (you can consider it as a side note error mentioning):

Error completing request
Arguments: ('', [], False, -1) {}
Traceback (most recent call last):
  File "/Users/hubert/stable-diffusion-webui/modules/call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "/Users/hubert/stable-diffusion-webui/modules/ui_common.py", line 46, in save_files
    data = json.loads(js_data)
  File "/opt/homebrew/Cellar/python@3.10/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/homebrew/Cellar/python@3.10/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/homebrew/Cellar/python@3.10/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The settings for the image generation and upscaling were as follows:

Checkpoint/model: f222.ckpt [9e2c6ceff3] Prompt (I hope it will not sound offensive): naked woman on her knees, with large breasts, full body view, face pointing at the camera, outdoors on a sunny, cloudless day (although I assume that as far as the occurrence of the error itself is concerned, it does not matter what type of prompt is used) Negative prompt: It was left blank Sampling method: DPM++ 2M Karras Sampling steps: 30 Restore faces: Checked Tiling: Unchecked Hires. fix: Checked Resize: from 512x512 to 2048x2048 Upscaler: Latent Hires steps: 0 Denoising strength: 0.7 Upscale by: 4 Resize width: Inactive (set to 0) Resize height: Inactive (set to 0) Width: 512 Height: 512 Batch count: 1 Batch size: 1 CFG Scale: 7.5 Seed: -1 Script: None

I also don't use any extensions.

Hardware Specification:

Apple Mac Mini M2 Pro, with a 12-core CPU, an Integrated GPU, 32 GB of integrated RAM (utilized as VRAM), and an 8 TB SSD.

Software Specification:

MacOS Ventura 13.3.1 Stable Diffusion 22bcc7b

Steps to reproduce the problem

  1. Choose the f222.ckpt [9e2c6ceff3] checkpoint/model (if you don't have it, choose a different checkpoint/model of your choice).

  2. Enter the prompt of your choice.

  3. Leave the Negative prompt textbox blank.

  4. Select all the other settings as described at the bottom of the „What happened?” section.

  5. Click the „Generate” button.

  6. Inspect the Terminal console/the command line manager console that you have and use for any Error: command buffer exited with error status. and RuntimeError: Invalid buffer size: errors.

What should have happened?

Normally, at the process of simultaneously generating an image at the 512x512 resolution, and upscaling it to the 2048x2048 resolution, eventually, the final image of the 2048x2048 resolution should be generated, either with no errors whatsoever occurring in the process, or at least with such errors occurring in the process that would not result in the final image not being generated at all.

Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/22bcc7b

What platforms do you use to access the UI ?

MacOS

What browsers do you use to access the UI ?

Apple Safari

Command Line Arguments

No

List of extensions

No

Console logs

Last login: Sun Apr 30 13:51:50 on ttys000
hubert@Huberts-Mac-Mini ~ % cd stable-diffusion-webui 
hubert@Huberts-Mac-Mini stable-diffusion-webui % bash webui.sh

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on hubert user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Python 3.10.11 (main, Apr  7 2023, 07:24:53) [Clang 14.0.0 (clang-1400.0.29.202)]
Commit hash: 22bcc7be428c94e9408f589966c2040187245d81
Installing requirements for Web UI
Launching Web UI with arguments: --skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
No module 'xformers'. Proceeding without it.
==============================================================================
You are running torch 1.12.1.
The program is tested to work with torch 1.13.1.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
Loading weights [cc6cb27103] from /Users/hubert/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.ckpt
Creating model from config: /Users/hubert/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying cross attention optimization (InvokeAI).
Textual inversion embeddings loaded(1): Bare Male Feet
Model loaded in 2.4s (load weights from disk: 0.6s, create model: 0.6s, apply weights to model: 0.4s, apply half(): 0.3s, move model to device: 0.4s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 5.1s (import torch: 0.8s, import gradio: 0.6s, import ldm: 0.2s, other imports: 0.6s, load scripts: 0.3s, load SD checkpoint: 2.5s, create ui: 0.1s).
Loading weights [9e2c6ceff3] from /Users/hubert/stable-diffusion-webui/models/Stable-diffusion/f222.ckpt
Applying cross attention optimization (InvokeAI).
Weights loaded in 1.7s (load weights from disk: 0.9s, apply weights to model: 0.4s, move model to device: 0.4s).
0%|                                                    | 0/30 [00:00<?, ?it/s]Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d2ab5500>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b14543e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
  3%|█▎                                       | 1/30 [03:00<1:27:17, 180.61s/it]Error: command buffer exited with error status. | 31/60 [07:13<26:24, 54.63s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc21a10>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
  7%|██▋                                      | 2/30 [05:59<1:23:53, 179.78s/it]Error: command buffer exited with error status. | 32/60 [10:12<42:56, 92.00s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2adfa2bb0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aae16bd0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af8919b0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 10%|████                                     | 3/30 [08:56<1:20:14, 178.30s/it]Error: command buffer exited with error status.| 33/60 [13:09<52:48, 117.37s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d24a3e20>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d24afe60>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2ba02e720>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abdbea90>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x11c7e4640>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d24823c0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b9f66110>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 17%|██████▊                                  | 5/30 [15:04<1:15:40, 181.62s/it]Error: command buffer exited with error status.35/60 [19:17<1:02:59, 151.20s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af2d93a0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 30%|████████████▎                            | 9/30 [27:04<1:03:06, 180.31s/it]Error: command buffer exited with error status.39/60 [31:17<1:00:34, 173.05s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2dff55dc0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 33%|██████████████                            | 10/30 [30:04<59:58, 179.93s/it]Error: command buffer exited with error status.| 40/60 [34:16<58:17, 174.87s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af2b54e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20999d0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af263b00>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2ae7f0a20>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 37%|███████████████▍                          | 11/30 [33:07<57:18, 180.99s/it]Error: command buffer exited with error status.| 41/60 [37:20<56:10, 177.42s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x11c7e0460>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107cfb930>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107b72b70>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 40%|████████████████▊                         | 12/30 [36:03<53:52, 179.58s/it]Error: command buffer exited with error status.| 42/60 [40:16<53:07, 177.11s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc1c620>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d45339b0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b205e310>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 43%|██████████████████▏                       | 13/30 [38:57<50:21, 177.73s/it]Error: command buffer exited with error status.| 43/60 [43:10<49:52, 176.02s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d45c1d10>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20232c0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2dff24640>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b2043860>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b2049b00>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 47%|███████████████████▌                      | 14/30 [41:48<46:50, 175.64s/it]Error: command buffer exited with error status.| 44/60 [46:00<46:31, 174.45s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d45926f0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af80c650>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 50%|█████████████████████                     | 15/30 [44:42<43:47, 175.18s/it]Error: command buffer exited with error status.| 45/60 [48:55<43:35, 174.35s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b91c7540>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d456aaa0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 53%|██████████████████████▍                   | 16/30 [47:35<40:44, 174.59s/it]Error: command buffer exited with error status.| 46/60 [51:48<40:36, 174.01s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107cd6aa0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b203e120>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x356d4ca90>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2dff91bf0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2dffc3ff0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 57%|███████████████████████▊                  | 17/30 [50:27<37:41, 173.94s/it]Error: command buffer exited with error status.| 47/60 [54:40<37:36, 173.54s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d456acb0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x356d45670>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107b72ad0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aefc99e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aefab810>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 60%|█████████████████████████▏                | 18/30 [53:21<34:45, 173.77s/it]Error: command buffer exited with error status.| 48/60 [57:34<34:41, 173.49s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x11c7e02e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d45a0680>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 63%|██████████████████████████▌               | 19/30 [56:12<31:43, 173.01s/it]Error: command buffer exited with error status.49/60 [1:00:25<31:40, 172.82s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2ae806900>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d4566960>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc1a6b0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af5eb780>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2af5ed3f0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20504e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107bcb970>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 67%|████████████████████████████              | 20/30 [59:02<28:41, 172.20s/it]Error: command buffer exited with error status.50/60 [1:03:15<28:40, 172.06s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b743dc00>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20375f0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20293b0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aefa72e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 70%|████████████████████████████            | 21/30 [1:01:52<25:42, 171.43s/it]Error: command buffer exited with error status.51/60 [1:06:05<25:41, 171.33s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abdfc9f0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aefb3ba0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc1fb10>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d45977c0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 73%|█████████████████████████████▎          | 22/30 [1:04:41<22:45, 170.75s/it]Error: command buffer exited with error status.52/60 [1:08:54<22:45, 170.68s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aefd5390>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d457d0e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 77%|██████████████████████████████▋         | 23/30 [1:07:32<19:56, 170.96s/it]Error: command buffer exited with error status.53/60 [1:11:45<19:56, 170.92s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20355f0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d4549220>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 80%|████████████████████████████████        | 24/30 [1:10:20<16:59, 169.90s/it]Error: command buffer exited with error status.54/60 [1:14:33<16:59, 169.87s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abdd8fc0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc15c90>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b204bda0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 83%|█████████████████████████████████▎      | 25/30 [1:13:06<14:04, 168.86s/it]Error: command buffer exited with error status.55/60 [1:17:19<14:04, 168.84s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2aefa5460>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d455e4e0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 87%|██████████████████████████████████▋     | 26/30 [1:15:57<11:17, 169.47s/it]Error: command buffer exited with error status.56/60 [1:20:10<11:17, 169.44s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107b70d40>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2abc19490>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d456c5f0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 90%|████████████████████████████████████    | 27/30 [1:18:46<08:27, 169.17s/it]Error: command buffer exited with error status.57/60 [1:22:59<08:27, 169.16s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2b20aeb40>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d4569fa0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x11c7e2c60>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107cf7460>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 93%|█████████████████████████████████████▎  | 28/30 [1:21:38<05:40, 170.04s/it]Error: command buffer exited with error status.58/60 [1:25:51<05:40, 170.04s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x356d3fbc0>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
 97%|██████████████████████████████████████▋ | 29/30 [1:24:28<02:50, 170.08s/it]Error: command buffer exited with error status.59/60 [1:28:41<02:50, 170.08s/it]
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x2d456c260>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
Error: command buffer exited with error status.
    The Metal Performance Shaders operations encoded on it may not have completed.
    Error: 
    (null)
    Internal Error (0000000e:Internal Error)
    <AGXG14XFamilyCommandBuffer: 0x107bcc570>
    label = <none> 
    device = <AGXG14SDevice: 0x29487d400>
        name = Apple M2 Pro 
    commandQueue = <AGXG14XFamilyCommandQueue: 0x29488c200>
        label = <none> 
        device = <AGXG14SDevice: 0x29487d400>
            name = Apple M2 Pro 
    retainedReferences = 1
100%|████████████████████████████████████████| 30/30 [1:27:18<00:00, 174.63s/it]
Error completing request█████████████████████| 60/60 [1:31:31<00:00, 170.18s/it]
Arguments: ('task(lixqsqhj8ad9gxz)', 'naked woman on her knees, with large breasts, full body view, face pointing at the camera, outdoors on a sunny, cloudless day', '', [], 30, 15, True, False, 1, 1, 7.5, -1.0, -1.0, 0, 0, 0, False, 512, 512, True, 0.7, 4, 'Latent', 0, 0, 0, [], 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "/Users/hubert/stable-diffusion-webui/modules/call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "/Users/hubert/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/Users/hubert/stable-diffusion-webui/modules/txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "/Users/hubert/stable-diffusion-webui/modules/processing.py", line 503, in process_images
    res = process_images_inner(p)
  File "/Users/hubert/stable-diffusion-webui/modules/processing.py", line 655, in process_images_inner
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "/Users/hubert/stable-diffusion-webui/modules/processing.py", line 655, in <listcomp>
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "/Users/hubert/stable-diffusion-webui/modules/processing.py", line 440, in decode_first_stage
    x = model.decode_first_stage(x)
  File "/Users/hubert/stable-diffusion-webui/modules/sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "/Users/hubert/stable-diffusion-webui/modules/sd_hijack_utils.py", line 26, in __call__
    return self.__sub_func(self.__orig_func, *args, **kwargs)
  File "/Users/hubert/stable-diffusion-webui/modules/sd_hijack_unet.py", line 76, in <lambda>
    first_stage_sub = lambda orig_func, self, x, **kwargs: orig_func(self, x.to(devices.dtype_vae), **kwargs)
  File "/Users/hubert/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/hubert/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 826, in decode_first_stage
    return self.first_stage_model.decode(z)
  File "/Users/hubert/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/autoencoder.py", line 90, in decode
    dec = self.decoder(z)
  File "/Users/hubert/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/hubert/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/model.py", line 631, in forward
    h = self.mid.attn_1(h)
  File "/Users/hubert/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/hubert/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/model.py", line 191, in forward
    w_ = torch.bmm(q,k)     # b,hw,hw    w[b,i,j]=sum_c q[b,i,c]k[b,c,j]
RuntimeError: Invalid buffer size: 16.00 GB

Additional information

No response

Smilecat1202 commented 1 year ago

Has the problem been solved? I also encountered this problem