search

Bing-su / adetailer

Auto detecting, masking and inpainting with detection model.
GNU Affero General Public License v3.0
4.08k stars 315 forks source link

[Bug]: ADetailer crashes with some mediapipe models in webui v1.10.1 #671

Closed tndgpp closed 4 weeks ago

tndgpp commented 1 month ago

Describe the bug

Running AUTOMATIC1111 version: [v1.10.1]  •  python: 3.11.2  •  torch: 2.1.2+cu121  •  xformers: 0.0.23.post1  •  gradio: 3.41.2 ADetailer searches the element defined in the model of the first enabled tab, finds the element, refines it but then crashes, failing to inpaint the main image

Steps to reproduce

I wrote a simple prompt ("beautiful girl standing in a city street, intricate detail") with no negative, enabled three ADetailer tabs (face_yolov8n.pt, mediapipe_face_mesh_eyes_only, hand_yolov8n.pt in this order), and started the image generation. The image was generated, but ADetailer crashed while processing first tab.

Screenshots

No response

Console logs, from start to end.

0: 640x448 1 face, 53.7ms
Speed: 2.0ms preprocess, 53.7ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)
WARNING:root:Sampler Scheduler autocorrection: "DPM++ 2M" -> "DPM++ 2M", "None" -> "Automatic"
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:03<00:00,  4.49it/s]
*** Error running postprocess_image: /opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/scripts/!adetailer.py
    Traceback (most recent call last):
      File "/opt/AUTOMATIC1111/stable-diffusion-webui/modules/scripts.py", line 912, in postprocess_image
        script.postprocess_image(p, pp, *script_args)
      File "/opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/aaaaaa/traceback.py", line 173, in wrapper
        raise error from None
    RuntimeError: 
    ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
    │                                                        System info                                                                                                                                                                                                                                                      │
    │ ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                                                                                                                                                                              │
    │ ┃             ┃ Value                                                                                                    ┃                                                                                                                                                                                              │
    │ ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩                                                                                                                                                                                              │
    │ │    Platform │ Linux-6.1.0-23-amd64-x86_64-with-glibc2.36                                                               │                                                                                                                                                                                              │
    │ │      Python │ 3.11.2 (main, May  2 2024, 11:59:08) [GCC 12.2.0]                                                        │                                                                                                                                                                                              │
    │ │     Version │ v1.10.1                                                                                                  │                                                                                                                                                                                              │
    │ │      Commit │ 82a973c04367123ae98bd9abdf80d9eda9b910e2                                                                 │                                                                                                                                                                                              │
    │ │ Commandline │ ['launch.py', '--medvram', '--xformers', '--disable-safe-unpickle']                                      │                                                                                                                                                                                              │
    │ │   Libraries │ {'torch': '2.1.2+cu121', 'torchvision': '0.16.2+cu121', 'ultralytics': '8.2.16', 'mediapipe': '0.10.14'} │                                                                                                                                                                                              │
    │ └─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┘                                                                                                                                                                                              │
    │                                              Inputs                                                                                                                                                                                                                                                                     │
    │ ┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                                                                                                                                                                                                       │
    │ ┃                 ┃ Value                                                                       ┃                                                                                                                                                                                                                       │
    │ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩                                                                                                                                                                                                                       │
    │ │          prompt │ beautiful girl standing in a city street, intricate details                 │                                                                                                                                                                                                                       │
    │ │ negative_prompt │                                                                             │                                                                                                                                                                                                                       │
    │ │          n_iter │ 1                                                                           │                                                                                                                                                                                                                       │
    │ │      batch_size │ 1                                                                           │                                                                                                                                                                                                                       │
    │ │           width │ 512                                                                         │                                                                                                                                                                                                                       │
    │ │          height │ 768                                                                         │                                                                                                                                                                                                                       │
    │ │    sampler_name │ DPM++ 2M                                                                    │                                                                                                                                                                                                                       │
    │ │       enable_hr │ False                                                                       │                                                                                                                                                                                                                       │
    │ │     hr_upscaler │ Latent                                                                      │                                                                                                                                                                                                                       │
    │ │      checkpoint │ Realistic Models/CyberRealistic/cyberrealistic_v50.safetensors [32baa6e527] │                                                                                                                                                                                                                       │
    │ │             vae │ vae-ft-mse-840000-ema-pruned.safetensors                                    │                                                                                                                                                                                                                       │
    │ │            unet │ Automatic                                                                   │                                                                                                                                                                                                                       │
    │ └─────────────────┴─────────────────────────────────────────────────────────────────────────────┘                                                                                                                                                                                                                       │
    │                 ADetailer                                                                                                                                                                                                                                                                                               │
    │ ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓                                                                                                                                                                                                                                                                               │
    │ ┃                     ┃ Value           ┃                                                                                                                                                                                                                                                                               │
    │ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩                                                                                                                                                                                                                                                                               │
    │ │             version │ 24.6.0          │                                                                                                                                                                                                                                                                               │
    │ │            ad_model │ face_yolov8n.pt │                                                                                                                                                                                                                                                                               │
    │ │           ad_prompt │                 │                                                                                                                                                                                                                                                                               │
    │ │  ad_negative_prompt │                 │                                                                                                                                                                                                                                                                               │
    │ │ ad_controlnet_model │ None            │                                                                                                                                                                                                                                                                               │
    │ │              is_api │ False           │                                                                                                                                                                                                                                                                               │
    │ └─────────────────────┴─────────────────┘                                                                                                                                                                                                                                                                               │
    │ ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/aaaaaa/traceback.py:153 in        │                                                                                                                                                                                                                    │
    │ │ wrapper                                                                                          │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │   152 │   │   try:                                                                               │                                                                                                                                                                                                                    │
    │ │ ❱ 153 │   │   │   return func(*args, **kwargs)                                                   │                                                                                                                                                                                                                    │
    │ │   154 │   │   except Exception as e:                                                             │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/scripts/!adetailer.py:818 in      │                                                                                                                                                                                                                    │
    │ │ postprocess_image                                                                                │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │    817 │   │   │   │   │   continue                                                              │                                                                                                                                                                                                                    │
    │ │ ❱  818 │   │   │   │   is_processed |= self._postprocess_image_inner(p, pp, args, n=n)           │                                                                                                                                                                                                                    │
    │ │    819                                                                                           │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/scripts/!adetailer.py:739 in      │                                                                                                                                                                                                                    │
    │ │ _postprocess_image_inner                                                                         │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │    738 │   │   with change_torch_load():                                                         │                                                                                                                                                                                                                    │
    │ │ ❱  739 │   │   │   pred = predictor(ad_model, pp.image, args.ad_confidence, **kwargs)            │                                                                                                                                                                                                                    │
    │ │    740                                                                                           │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/adetailer/mediapipe.py:24 in      │                                                                                                                                                                                                                    │
    │ │ mediapipe_predict                                                                                │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │    23 │   │   func = mapping[model_type]                                                         │                                                                                                                                                                                                                    │
    │ │ ❱  24 │   │   return func(image, confidence)                                                     │                                                                                                                                                                                                                    │
    │ │    25 │   msg = f"[-] ADetailer: Invalid mediapipe model type: {model_type}, Available: {list(   │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/extensions/adetailer/adetailer/mediapipe.py:130 in     │                                                                                                                                                                                                                    │
    │ │ mediapipe_face_mesh_eyes_only                                                                    │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │   129 │                                                                                          │                                                                                                                                                                                                                    │
    │ │ ❱ 130 │   with mp_face_mesh.FaceMesh(                                                            │                                                                                                                                                                                                                    │
    │ │   131 │   │   static_image_mode=True, max_num_faces=20, min_detection_confidence=confidence      │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/venv/lib/python3.11/site-packages/mediapipe/python/sol │                                                                                                                                                                                                                    │
    │ │ utions/face_mesh.py:95 in __init__                                                               │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │    94 │   """                                                                                    │                                                                                                                                                                                                                    │
    │ │ ❱  95 │   super().__init__(                                                                      │                                                                                                                                                                                                                    │
    │ │    96 │   │   binary_graph_path=_BINARYPB_FILE_PATH,                                             │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │ /opt/AUTOMATIC1111/stable-diffusion-webui/venv/lib/python3.11/site-packages/mediapipe/python/sol │                                                                                                                                                                                                                    │
    │ │ ution_base.py:248 in __init__                                                                    │                                                                                                                                                                                                                    │
    │ │                                                                                                  │                                                                                                                                                                                                                    │
    │ │   247 │                                                                                          │                                                                                                                                                                                                                    │
    │ │ ❱ 248 │   self._graph = calculator_graph.CalculatorGraph(                                        │                                                                                                                                                                                                                    │
    │ │   249 │   │   graph_config=canonical_graph_config_proto)                                         │                                                                                                                                                                                                                    │
    │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                                                                                                                                    │
    │ RuntimeError: ValidatedGraphConfig Initialization failed.                                                                                                                                                                                                                                                               │
    │ ConstantSidePacketCalculator: ; RET_CHECK failure (mediapipe/calculators/core/constant_side_packet_calculator.cc:65) (cc->OutputSidePackets().NumEntries(kPacketTag))==(options.packet_size())Number of output side packets has to be same as number of packets configured in options.                                  │
    │ ConstantSidePacketCalculator: ; RET_CHECK failure (mediapipe/calculators/core/constant_side_packet_calculator.cc:65) (cc->OutputSidePackets().NumEntries(kPacketTag))==(options.packet_size())Number of output side packets has to be same as number of packets configured in options.                                  │
    │ ImageToTensorCalculator: ; RET_CHECK failure (mediapipe/calculators/tensor/image_to_tensor_calculator.cc:144) ValidateOptionOutputDims(options) returned INTERNAL: ; RET_CHECK failure (./mediapipe/calculators/tensor/image_to_tensor_utils.h:134) options.has_output_tensor_float_range() ||                          │
    │ options.has_output_tensor_int_range() || options.has_output_tensor_uint_range()Output tensor range is required.                                                                                                                                                                                                         │
    │ SplitTensorVectorCalculator: The number of output streams should match the number of ranges specified in the CalculatorOptions.                                                                                                                                                                                         │
    │ SplitTensorVectorCalculator: The number of output streams should match the number of ranges specified in the CalculatorOptions.                                                                                                                                                                                         │
    │ SplitTensorVectorCalculator: The number of output streams should match the number of ranges specified in the CalculatorOptions.                                                                                                                                                                                         │
    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

---

List of installed extensions

ADetailer

dosubot[bot] commented 1 month ago

Hey there, @tndgpp! I'm here to help you out with any bugs, questions, or contributions you have. Let me know how I can assist you!

The crash during the processing of the first enabled tab when running ADetailer on AUTOMATIC1111 version v1.10.1 is likely due to compatibility issues. ADetailer does not support WebUI versions below 1.6.0, and there are specific imports and checks for compatibility with versions up to 1.9.0. If there are significant changes or updates in version v1.10.1 that are not compatible with ADetailer, it could cause the extension to crash [1][2].

Additionally, the args.py file for ADetailer defines a comprehensive list of arguments and configurations that the extension uses. If any of these configurations are not properly handled or if there are discrepancies in how these arguments are processed in version v1.10.1, it could lead to crashes [3].

To address this issue, you may need to review and update the handling of specific arguments and configurations to ensure compatibility with version v1.10.1.

To continue talking to Dosu, mention @dosu.

tndgpp commented 1 month ago

@dosu After some testing, I realized that the error is triggered if mediapipe_face_mesh and mediapipe_face_mesh_eyes_only are among the enabled models. In this case, adetailer crashes as soon as it tries to use one of these models, otherwise everything runs smoothly. I don't know if this is relevant in general but this is what happens in my case.

dosubot[bot] commented 1 month ago

To resolve the crash with ADetailer on AUTOMATIC1111 version v1.10.1 when the mediapipe_face_mesh and mediapipe_face_mesh_eyes_only templates are enabled, ensure that the mediapipe library is correctly imported and used in these functions. Here is the relevant part of the code that handles these templates:

def mediapipe_face_mesh(
    image: Image.Image, confidence: float = 0.3
) -> PredictOutput[int]:
    import mediapipe as mp

    mp_face_mesh = mp.solutions.face_mesh
    draw_util = mp.solutions.drawing_utils
    drawing_styles = mp.solutions.drawing_styles

    w, h = image.size

    with mp_face_mesh.FaceMesh(
        static_image_mode=True, max_num_faces=20, min_detection_confidence=confidence
    ) as face_mesh:
        arr = np.array(image)
        pred = face_mesh.process(arr)

        if pred.multi_face_landmarks is None:
            return PredictOutput()

        preview = arr.copy()
        masks = []

        for landmarks in pred.multi_face_landmarks:
            draw_util.draw_landmarks(
                image=preview,
                landmark_list=landmarks,
                connections=mp_face_mesh.FACEMESH_TESSELATION,
                landmark_drawing_spec=None,
                connection_drawing_spec=drawing_styles.get_default_face_mesh_tesselation_style(),
            )

            points = np.array(
                [[land.x * w, land.y * h] for land in landmarks.landmark], dtype=int
            )
            outline = cv2.convexHull(points).reshape(-1).tolist()

            mask = Image.new("L", image.size, "black")
            draw = ImageDraw.Draw(mask)
            draw.polygon(outline, fill="white")
            masks.append(mask)

        bboxes = create_bbox_from_mask(masks, image.size)
        preview = Image.fromarray(preview)
        return PredictOutput(bboxes=bboxes, masks=masks, preview=preview)

def mediapipe_face_mesh_eyes_only(
    image: Image.Image, confidence: float = 0.3
) -> PredictOutput[int]:
    import mediapipe as mp

    mp_face_mesh = mp.solutions.face_mesh

    left_idx = np.array(list(mp_face_mesh.FACEMESH_LEFT_EYE)).flatten()
    right_idx = np.array(list(mp_face_mesh.FACEMESH_RIGHT_EYE)).flatten()

    w, h = image.size

    with mp_face_mesh.FaceMesh(
        static_image_mode=True, max_num_faces=20, min_detection_confidence=confidence
    ) as face_mesh:
        arr = np.array(image)
        pred = face_mesh.process(arr)

        if pred.multi_face_landmarks is None:
            return PredictOutput()

        preview = image.copy()
        masks = []

        for landmarks in pred.multi_face_landmarks:
            points = np.array(
                [[land.x * w, land.y * h] for land in landmarks.landmark], dtype=int
            )
            left_eyes = points[left_idx]
            right_eyes = points[right_idx]
            left_outline = cv2.convexHull(left_eyes).reshape(-1).tolist()
            right_outline = cv2.convexHull(right_eyes).reshape(-1).tolist()

            mask = Image.new("L", image.size, "black")
            draw = ImageDraw.Draw(mask)
            for outline in (left_outline, right_outline):
                draw.polygon(outline, fill="white")
            masks.append(mask)

        bboxes = create_bbox_from_mask(masks, image.size)
        preview = draw_preview(preview, bboxes, masks)
        return PredictOutput(bboxes=bboxes, masks=masks, preview=preview)

Ensure that the mediapipe library is installed in your environment:

pip install mediapipe

If the crash persists, it might be due to other issues not covered in the provided context [1].

To continue talking to Dosu, mention @dosu.

tndgpp commented 1 month ago

@dosu mediapipe library is actually installed (version 0.10.14)

$ pip show mediapipe
Name: mediapipe
Version: 0.10.14
Summary: MediaPipe is the simplest way for researchers and developers to build world-class ML solutions and applications for mobile, edge, cloud and the web.
Home-page: https://github.com/google/mediapipe
Author: The MediaPipe Authors
Author-email: mediapipe@google.com
License: Apache 2.0
Location: /opt/AUTOMATIC1111/stable-diffusion-webui/venv/lib/python3.11/site-packages
Requires: absl-py, attrs, flatbuffers, jax, jaxlib, matplotlib, numpy, opencv-contrib-python, protobuf, sounddevice
Required-by: handrefinerportable
tndgpp commented 4 weeks ago

With version 24.8.0, adetailer does not crash anymore when encountering those mediapipe models, it simply does not detect anything and proceeds with the next tab enabled (if any). So, this specific bug can be considered solved. However, the author should clarify whether those mediapipe models should be considered deprecated or not.