how to detect and read monochrome images

[tonycoz]

libheif now allows writing to monochrome/grayscale images, but the provided API doesn't seem to provide a way to detect if the input image is monochrome. This is reflected in the sample code, for example:

$ file test.jpg 
test.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 150x150, components 1
$ ~/local/libheif-1.12.0/bin/heif-enc -o test.heif test.jpg
$ ~/local/libheif-1.12.0/bin/heif-convert test.heif test-out.jpg
File contains 1 images
Written to test-out.jpg
$ file test-out.jpg 
test-out.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 150x150, components 3

Note how the number of components has changed from 1 to 3.

Unfortunately I don't have all of the relevant HEIF specs, so I'm not sure which boxes would need to be used to determine this.

I suspect the pixi box given it changes from (8,8,8) to (8) going from an RGB to a monochrome image, but I suspect hvcC also includes relevant information.

[silverbacknet]

The resulting image is going to be full color, but the U/V planes will be entirely zero (hopefully). Unless you plumb quite deep into the API, everything is converted to RGB and then converted back to YUV for processing; this is partly because JPEG can have 1x2 subsampling, which neither HEVC nor AV1 can support, and partly because it legitimately just makes everything much simpler.

heif-enc does not support creating single-plane images, although it can read them just fine.

[tonycoz]

The code does appear to support Y only images on read, but it's not exposed via heif_image_handles, If I do:

    struct heif_image *him = NULL;

    err = heif_decode_image(img_handle, &him, heif_colorspace_undefined,
                            heif_chroma_undefined, NULL);
    if (err.code == heif_error_Ok) {
      fprintf(stderr, "decoded cs %d chroma %d\n", heif_image_get_colorspace(him),
              heif_image_get_chroma_format(him));
      fprintf(stderr, "y %d Cb %d Cr %d\n", heif_image_has_channel(him, heif_channel_Y),
              heif_image_has_channel(him, heif_channel_Cb), heif_image_has_channel(him, heif_channel_Cr));
      heif_image_release(him);
    }

on a monochrome image (the result from heif-enc of the grayscale JPEG), I get:

decoded cs 2 chroma 0
y 1 Cb 0 Cr 0

cs 2 corresponds to heif_colorspace_monochrome, chroma 0 corresponds to heif_chroma_monochrome and it has only the Y plane as expected.

I'd like to avoid the cost of decoding twice, the library appears to have the information needed, but doesn't expose it via heif_image_handle.

Starting from a similar RGB JPEG file I get:

decoded cs 1 chroma 3
y 0 Cb 0 Cr 0

which corresponds to heif_colorspace_RGB and heif_chroma_444 and reasonably doesn't have any of the YCbCr planes.

[farindk]

Determining the output color space can be more complicated because the image can in fact be composed of several parts with different color spaces.

What would be the use case of querying the color space before decoding? Why isn't it sufficient to decode with heif_colorspace_undefined and check the output?

[tonycoz]

There's no documentation so I can't tell what possible colourspaces heif_decode_image() can return, if it's only RGB and monochrome, then I don't need this, but if it can return YCbCr then I'd like be able to tell before the presumably expensive step of decoding so I can ask for RGB.

Currently I do that expensive step - I ask for an image, if it's monochrome or RGB fine, if it's YCbCr I have to delete the image and explicitly ask for RGB.

I'd like to avoid that double-decode, and be able to distinguish between monochrome and colour images.

[silverbacknet]

Documentation could be improved, for sure.

Simply using your existing pipeline, if you get a YCbCr image back, you can feed it to convert_colorspace() (see the end of heif_colorconversion.h for args) to get that RGB without the whole rigmorale of re-decoding it. Or don't even check and convert_colorspace() will always return your format or choice, with no conversion cost if it's already in that format. Or just always call heif_decode with heif_colorspace_RGB, which does the above for you.

The possible colorspaces are the same as what you can feed, and if you go with making everything into a single format, you probably want to specify the exact chroma format you expect as well:

enum heif_chroma
{
  heif_chroma_undefined = 99,
  heif_chroma_monochrome = 0,
  heif_chroma_420 = 1,
  heif_chroma_422 = 2,
  heif_chroma_444 = 3,
  heif_chroma_interleaved_RGB = 10,
  heif_chroma_interleaved_RGBA = 11,
  heif_chroma_interleaved_RRGGBB_BE = 12,
  heif_chroma_interleaved_RRGGBBAA_BE = 13,
  heif_chroma_interleaved_RRGGBB_LE = 14,
  heif_chroma_interleaved_RRGGBBAA_LE = 15
};
enum heif_colorspace
{
  heif_colorspace_undefined = 99,
  heif_colorspace_YCbCr = 0,
  heif_colorspace_RGB = 1,
  heif_colorspace_monochrome = 2
};

I have no idea if you can stick an XYZ or CMYK in heif (and have it actually tagged as such), but I suppose that would probably return heif_colorspace_undefined.

[bigcat88]

So there is no an easy way to detect what resulting image format will be without decoding? I want to add ability for python bindings to open images in monochrome mode, but I do not want to get YCBCR as a resulting mode when setting heif_colorspace_undefined Only monochrome, monochrome with alpha or rgb(a)

[bigcat88]

this can be closed, starting from libheif 1.17.0 there is heif_image_handle_get_preferred_decoding_colorspace function that is working fine for this case