[InferenceSlicer] - instance segmentation inference results are not post-processed properly

[SaiJeevanPuchakayala]

I am experiencing an issue where the model is performing inference on the full image resolution of 2048x2048 instead of the sliced resolution of 512x512 as intended. Below are the details of the function and the problem encountered.

def callback(image: np.ndarray) -> sv.Detections:
    print(f"Image shape in callback: {image.shape}")  # Debugging line
    result = model(image, device='cuda')[0]
    return sv.Detections.fromultralytics(result)

# Function to process an image and display the result
def process_image(image_path: str):
    try:
        image = cv2.imread(image_path)
        print("Input image resolution", image.shape)
        if image is None:
            raise FileNotFoundError(f"Image at path {image_path} could not be loaded.")

        # Perform inference
        slicer = sv.InferenceSlicer(
            slice_wh=(512, 512),
            overlap_ratio_wh=(0.2, 0.2),
            thread_workers=10,
            iou_threshold=0.5,
            overlap_filter_strategy=sv.OverlapFilter.NON_MAX_MERGE,
            callback=callback
        )

        detections = slicer(image)

        # Filter detections based on confidence
        detections = detections[detections.confidence > 0.3]

        mask_annotator = sv.MaskAnnotator()
        annotated_image = mask_annotator.annotate(scene=image.copy(), detections=detections)

        # Draw contours around the masks and fit ellipses
        for i in range(detections.mask.shape[0]):
            mask = detections.mask[i].astype(np.uint8)
            contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
            cv2.drawContours(annotated_image, contours, -1, (0, 255, 0), 2)

        plt.figure(figsize=(20, 20))
        plt.imshow(cv2.cvtColor(annotated_image, cv2.COLOR_BGR2RGB))
        plt.axis('off')
        plt.show()

        # Print the number of detections
        print(f"Number of detections: {len(detections.confidence)}")

        # Clear memory
        del slicer, detections, annotated_image
        torch.cuda.empty_cache()
        gc.collect()

    except Exception as e:
        print(f"An error occurred: {str(e)}")
        raise

# Example usage
process_image(image_path)

Output

Input image resolution (2048, 1448, 3)
Image shape in callback: (512, 512, 3)
...
0: 2048x2048 4 WholeRedRices, 8126.6ms
Speed: 34.0ms preprocess, 8126.6ms inference, 4.0ms postprocess per image at shape (1, 3, 2048, 2048)
Image shape in callback: (512, 512, 3)
...
0: 2048x2048 1 BrokenRedRice, 12 WholeRedRices, 7223.1ms
Speed: 120.0ms preprocess, 7223.1ms inference, 118.0ms postprocess per image at shape (1, 3, 2048, 2048)
Image shape in callback: (408, 218, 3)

Issue

The model is performing inference on the full image resolution (2048x2048) instead of the sliced resolution (512x512) as specified in the InferenceSlicer. This results in longer inference times and processing on larger image chunks than intended.

Steps to Reproduce

1. Use the provided function code.
2. Process an image with a resolution of 2048x2048.
3. Observe the output logs indicating the inference is happening on 2048x2048 instead of 512x512.

Expected Behavior

The model should perform inference on 512x512 slices of the image, as specified in the InferenceSlicer.

Actual Behavior

Inference is performed on the full 2048x2048 image resolution.

Environment

1. Model: Custom-trained on images with resolution 512 X 512 which are sliced from 2048 X 2048.
2. Device: CUDA-enabled GPU

Additional Context

Any insights or suggestions to ensure the model performs inference on the specified 512x512 slices would be greatly appreciated.

[SkalskiP]

Hi @SaiJeevanPuchakayala 👋🏻 Looks like ultralytics is resizing images before inference. This is probably because during training you passed imgsz = 2048 as argument. Try to update result = model(image, device='cuda')[0] to result = model(image, device='cuda', imgsz=640)[0], rerun the script.

[SaiJeevanPuchakayala]

Hi @SaiJeevanPuchakayala 👋🏻 Looks like ultralytics is resizing images before inference. This is probably because during training you passed imgsz = 2048 as argument. Try to update result = model(image, device='cuda')[0] to result = model(image, device='cuda', imgsz=640)[0], rerun the script.

Thanks for the fix @SkalskiP.

The line of code below is working for me.

result = model(image, device='cuda', imgsz=512)[0]

But while stitching back the image the 512 images to 2048, I'm not able to get a few detections and annotations properly near the sliced region as shown in the image below.

[SkalskiP]

@SaiJeevanPuchakayala, what overlap_ratio_wh did you use? (0.2, 0.2)? It looks like there is no overlap at all.

[SaiJeevanPuchakayala]

@SaiJeevanPuchakayala, what overlap_ratio_wh did you use? (0.2, 0.2)? It looks like there is no overlap at all.

@SkalskiP I actually used overlap_ratio_wh as (0.2, 0.2) both for training and inference and even played with it by increasing/decreasing it along with overlap filters (NMS, NMM, None), but still, the issue of lines in between a few detections and their corresponding segments while inference still exists as shown in the image below.

[SaiJeevanPuchakayala]

@SaiJeevanPuchakayala, what overlap_ratio_wh did you use? (0.2, 0.2)? It looks like there is no overlap at all.

@SkalskiP I actually used overlap_ratio_wh as (0.2, 0.2) both for training and inference and even played with it by increasing/decreasing it along with overlap filters (NMS, NMM, None), but still, the issue of lines in between a few detections and their corresponding segments while inference still exists as shown in the image below.

@SkalskiP I'm still facing the same issue. Is there any resolution for this that you can suggest?

[SkalskiP]

@SaiJeevanPuchakayala, have you tried playing with iou_threshold as well?

[SaiJeevanPuchakayala]

@SaiJeevanPuchakayala, have you tried playing with iou_threshold as well?

Yes @SkalskiP, I've tried that as well, no hope.

[SkalskiP]

@SaiJeevanPuchakayala, can you provide us with your model, image, and code?

[SaiJeevanPuchakayala]

@SaiJeevanPuchakayala, can you provide us with your model, image, and code?

Hi @SkalskiP,

Thank you for your continuous support. I have uploaded the model, image, and code to a Google Drive folder for your reference. You can access it here: https://drive.google.com/drive/folders/1nn08DGO7-I1rRX-5Czm_tFn7hWV5J9IN?usp=sharing

Here are some additional details about the model:

• The model is trained on YOLOv8 seg medium.
• The training was done using 512x512 sliced images.
• For inference, segmentation is performed on the sliced images split from the test image and then stitched together back to the original image.

Please have a look and let me know if you need any additional information.

[SkalskiP]

@SaiJeevanPuchakayala, realistically, I won't be able to take a deeper look into it. Sorry, I have too much work to do with the upcoming YT video. This would need to wait for @LinasKo to get back next week. Maybe @onuralpszr or @hardikdava have some time to take a look into it?

[onuralpszr]

@SkalskiP sure, I am going to take look let me setup in collab and start playing with it.

[SkalskiP]

@onuralpszr, you are the GOAT! 🐐

[onuralpszr]

@onuralpszr, you are the GOAT! 🐐

Likewise you too :) Let me do some testing get back to comments with my findings

[SkalskiP]

I'll update the name of the issue to better reflect what's going on.

[SaiJeevanPuchakayala]

@SkalskiP sure, I am going to take look let me setup in collab and start playing with it.

Hey Hi @onuralpszr 👋, have you got time to do some testing?

[onuralpszr]

@SkalskiP sure, I am going to take look let me setup in collab and start playing with it.

Hey Hi @onuralpszr 👋, have you got time to do some testing?

Yes, I am looking and let me bit more test and will come back to you, I had some busy tasks I had handle as well, sorry for bit delay. I will come back to you

[SkalskiP]

Hi @SaiJeevanPuchakayala and @onuralpszr, have you managed to track down the problem?

[SaiJeevanPuchakayala]

Hi @SaiJeevanPuchakayala and @onuralpszr, have you managed to track down the problem?

Hey @SkalskiP, not yet, still facing the same issue with grains not being stiched back.

[onuralpszr]

Hi @SaiJeevanPuchakayala and @onuralpszr, have you managed to track down the problem?

I did stuck for a bit but made progress afterwards. I had a busy work week, let me handle tackle today and show some result and talk based on it. Sorry for delay.

[hardikdava]

@onuralpszr Let me know if you need extra pair of hands on this one.

[SkalskiP]

No worries, @onuralpszr ;) I'm just curious why it does not work as expected.

[SaiJeevanPuchakayala]

@onuralpszr Let me know if you need extra pair of hands on this one.

@onuralpszr and @hardikdava, have you guys got time to track down the problem?

[hardikdava]

Hey @onuralpszr, can you post your findings on this issue here? I can take a look later today. Thanks.