[Bug]: `PIL.Image` is 700+% faster than `fastdup_imread`

[dnth]

What happened?

fastdup uses a utility function fastdup_imread throughout the codebase. But I found that this function is very slow compared to using plain PIL.Image

My goal is to get a PIL Image from an image path.

• Using fastdup_imread

• Using PIL

That's a whopping 755.75% increase in run time when using fastdup_imread!

The difference is still large even when the codes to convert from BGR to RGB are commented out (243.08% increase).

I believe in a large dataset this difference will be significant.

What did you expect to see?

A smaller difference in run time for both.

What version of fastdup were you runnning on?

1.43

What version of Python were you running on?

Python 3.10

Operating System

Ubuntu 22.04

Reproduction steps

Run this Colab - https://colab.research.google.com/drive/1dXver-IrwMz4aBnkW2GSLo9gmHEIn2Iu?usp=sharing

Relevant log output

No response

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response

[dbickson]

HI @dnth I believe the [::,::,::] may slow down I think they are more efficient ways, you can do it with cv2 . Not sure if the pil image open actually reads the image or is it a lazy operation just opening a file handle?

[dbickson]

Take a look here and feel free to extend the func to support your needs

    if img1_path.lower().endswith('.heic') or img1_path.lower().endswith('.heif'):
        img = Image.open(img1_path)
        assert img is not None, f"Failed to open image from {img1_path}"
        img = np.array(img)
        channels = img.shape[-1] if img.ndim == 3 else 1
        if channels == 1:
            img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)
        elif channels == 4:
            img = cv2.cvtColor(img, cv2.COLOR_RGBA2RGB)
            img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
        else:
            img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
    else:
        img = cv2.imread(img1_path, cv2.IMREAD_UNCHANGED)
    assert img is not None, f"Failed to open image from {img1_path}"
    if img.dtype == 'uint16':
        img = cv2.normalize(img, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8U)
        channels = img.shape[-1] if img.ndim == 3 else 1
        if channels == 1:
            img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)
        elif channels == 4:
            img = cv2.cvtColor(img, cv2.COLOR_RGBA2RGB)
    return img

[dnth]

Thanks Danny. I ran a quick comparison using cv2. The cv2 run time fluctuates but always take longer than numpy img[:, :, ::-1]

[dnth]

To rule out the possibility of lazy execution, I displayed the image and made a comparison.

• Using fastdup_imread + numpy

• Using fastdup_imread + cv2

• Using PIL.Image