[Bug] Different type checking for same-size vs different size images in group

[kretes]

First of all - thanks for the lib, it is really nice and helpful!

I first came across a strange thing, which is a different working of the library when I pass data as imageIO images. Although they are numpy arrays - I think they aren't recognized correctly by the library, but this only affects the situation when the passed images are of diverse shapes.

See this reproducible example:

S1 = 300
S2 = 400
data1 = np.random.randint(0,255, (S1,S1,3))
imageio.imsave(img1_path := "/tmp/img1.png", data1)

data2 = np.random.randint(0,255, (S2,S2,3))
imageio.imsave(img2_path := "/tmp/img2.png", data2)

img = imageio.imread(img1_path)
img2 = imageio.imread(img2_path)

images = [img, img2]

for img in images:
    print(type(img), img.shape, img.size)

print(np.asarray(images).shape)
# images = [np.asarray(im) for im in images]

ipyplot.plot_class_tabs(images, ["t"]*2, max_imgs_per_tab=10, img_width=None, custom_texts=["d"]*2)

which just creates two random images and tries to display them with plot_class_tabs. The error got is :

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [73], in <module>
     17 print(np.asarray(images).shape)
     18 # images = [np.asarray(im) for im in images]
---> 20 ipyplot.plot_class_tabs(images, ["t"]*2, max_imgs_per_tab=10, img_width=None, custom_texts=["d"]*2)

File [...]/lib/python3.9/site-packages/ipyplot/_plotting.py:71, in plot_class_tabs(images, labels, custom_texts, max_imgs_per_tab, img_width, zoom_scale, force_b64, tabs_order)
     68 custom_texts = _np.asarray(custom_texts) if custom_texts is not None else custom_texts  # NOQA E501
     70 # run html helper function to generate html content
---> 71 html = _create_tabs(
     72     images=images,
     73     labels=labels,
     74     custom_texts=custom_texts,
     75     max_imgs_per_tab=max_imgs_per_tab,
     76     img_width=img_width,
     77     zoom_scale=zoom_scale,
     78     force_b64=force_b64,
     79     tabs_order=tabs_order)
     81 _display_html(html)

File [...]/lib/python3.9/site-packages/ipyplot/_html_helpers.py:130, in _create_tabs(images, labels, custom_texts, max_imgs_per_tab, img_width, zoom_scale, force_b64, tabs_order)
    127     active_tab = False
    129     tab_imgs_mask = labels == label
--> 130     html += _create_imgs_grid(
    131         images=images[tab_imgs_mask],
    132         labels=list(range(0, max_imgs_per_tab)),
    133         max_images=max_imgs_per_tab,
    134         img_width=img_width,
    135         zoom_scale=zoom_scale,
    136         custom_texts=custom_texts[tab_imgs_mask] if custom_texts is not None else None,  # NOQA E501
    137         force_b64=force_b64)
    139     html += '</div>'
    141 html += '</div>'

File [...]/lib/python3.9/site-packages/ipyplot/_html_helpers.py:359, in _create_imgs_grid(images, labels, custom_texts, max_images, img_width, zoom_scale, force_b64)
    356 html, grid_style_uuid = _get_default_style(img_width, zoom_scale)
    358 html += '<div id="ipyplot-imgs-container-div-%s">' % grid_style_uuid
--> 359 html += ''.join([
    360     _create_img(
    361         x, width=img_width, label=y,
    362         grid_style_uuid=grid_style_uuid,
    363         custom_text=text, force_b64=force_b64
    364     )
    365     for x, y, text in zip(
    366         images[:max_images], labels[:max_images],
    367         custom_texts[:max_images])
    368 ])
    369 html += '</div>'
    370 return html

File [...]/lib/python3.9/site-packages/ipyplot/_html_helpers.py:360, in <listcomp>(.0)
    356 html, grid_style_uuid = _get_default_style(img_width, zoom_scale)
    358 html += '<div id="ipyplot-imgs-container-div-%s">' % grid_style_uuid
    359 html += ''.join([
--> 360     _create_img(
    361         x, width=img_width, label=y,
    362         grid_style_uuid=grid_style_uuid,
    363         custom_text=text, force_b64=force_b64
    364     )
    365     for x, y, text in zip(
    366         images[:max_images], labels[:max_images],
    367         custom_texts[:max_images])
    368 ])
    369 html += '</div>'
    370 return html

File [...]/lib/python3.9/site-packages/ipyplot/_html_helpers.py:286, in _create_img(image, label, width, grid_style_uuid, custom_text, force_b64)
    283 # if image is not a string it means its either PIL.Image or np.ndarray
    284 # that's why it's necessary to use conversion to b64
    285 if use_b64:
--> 286     img_html += '<img src="data:image/png;base64,%s"/>' % _img_to_base64(image, width)  # NOQA E501
    288 html = """
    289 <div class="ipyplot-placeholder-div-%(0)s">
    290     <div id="ipyplot-content-div-%(0)s-%(1)s" class="ipyplot-content-div-%(0)s">
   (...)
    300 </div>
    301 """ % {'0': grid_style_uuid, '1': img_uuid, '2': label, '3': img_html}  # NOQA E501
    302 return html

File [...]/lib/python3.9/site-packages/ipyplot/_img_helpers.py:93, in _img_to_base64(image, target_width)
     91 # save image object to bytes stream
     92 output = io.BytesIO()
---> 93 image.save(output, format='PNG')
     94 # encode bytes as base64 string
     95 b64 = str(base64.b64encode(output.getvalue()).decode('utf-8'))

AttributeError: 'Array' object has no attribute 'save'

The reason for that is that I pass imageIO Arrays and not numpy arrays. IF any of those is changed in the example:

• set the same shape of all images (here set S1 = S2)
• convert all the arrays to native numpys (here - uncomment # images = [np.asarray(im) for im in images])

Everything works. I think the reason for that is that:

1. There is a conversion of all the images to a single numpy array in https://github.com/karolzak/ipyplot/blob/ff64f4a376e17bf86c6c6fb1d47ee99e50227708/ipyplot/_utils.py#L97 that works differently for same-shape input - for same shape we get a nparray of [N,W,H,C], but for a different-shape we get a numpy array of [N,] with objects inside, and the objects aren't then converted to numpy arrays but remain imageIO Arrays.
2. Some later code isn't checking the type in the right way: https://github.com/karolzak/ipyplot/blob/ff64f4a376e17bf86c6c6fb1d47ee99e50227708/ipyplot/_img_helpers.py#L77 - if we change type(img) == np.ndarray into isinstance(img, np.ndarray) - the subclass should be handled appropriately.

[karolzak]

Hi @kretes ! Thank you so much for investing your time in digging so deep into the source code to find the root cause and report it! Very much appreciated! I will try to recreate this problem following the details you shared and hopefully we can fix it with the next release. Thank you!