Reuse decoder buffer (instead of alloc + memcpy)

strukturag / libheif

libheif is an HEIF and AVIF file format decoder and encoder.

Other

1.71k stars 298 forks source link

Reuse decoder buffer (instead of alloc + memcpy) #1202

Open roticv opened 3 months ago

roticv commented 3 months ago

My understanding is that when attempting to decode heic to an output colorspace (such as RGB), libheif does this:

Use plugin to decode
Each plugin has its own decode buffer to decode to and call into heif_image_add_plane to allocate for each plane and then memcpy from the decode buffer into the heif image plane buffer
Color conversion (such as from YUV420 to RGB): another heif_image has to allocate buffer and the color conversion happens

Is there a need to memcpy from decoder buffer into heif image plane buffer? In theory, there isn't a need to do so, there is just a need to support custom memory (how to scan through the data for color conversion). This should improve heif decoding performance.

The above is just my surface understanding of libheif. Let me know if it makes sense.

farindk commented 3 months ago

There is no need to memcpy the decoded output into an heif_image. However, if we want to avoid this, the decoder library has to support a way to decode into externally provided memory, or it has to output the memory in a way such that ownership can be transferred. That's usually not the case.

roticv commented 3 months ago

However, if we want to avoid this, the decoder library has to support a way to decode into externally provided memory, or it has to output the memory in a way such that ownership can be transferred. That's usually not the case.

I am pretty sure this optimization is possible to experiment with libde265 (and its plugin in libheif).

farindk commented 3 months ago

However, if we want to avoid this, the decoder library has to support a way to decode into externally provided memory, or it has to output the memory in a way such that ownership can be transferred. That's usually not the case.

I am pretty sure this optimization is possible to experiment with libde265 (and its plugin in libheif).

I am pretty sure this is not possible out of the box (I also wrote libde265). But I could in theory add some API to libde265 to support this. However, I'm not sure this would make a big difference because the bottleneck is the color conversion and not a single image copy.

roticv commented 3 months ago

However, I'm not sure this would make a big difference because the bottleneck is the color conversion and not a single image copy.

Makes sense. I guess one possible area to look into is to have x86 implementation of color conversion