Idea: Vector and bitmap format mixing

PaintNinja commented 8 years ago

Hello, I've been watching FLIF development as a pro user and been really impressed with the progress being made.

I would like to share an idea which may improve compression ratio and possibly image quality too in some scenarios. I apologise in advance if this sounds like a bad idea, I don't have much knowledge in how image codecs work.

The idea is to mix both vector and bitmap graphics into a single image, allowing for improved compression ratios and possibly image quality in some scenarios. There are often solid shapes or gradients within images where they achieve much better compression as SVGZ files rather than PNG files, but could have more complex scenes in the same image which couldn't be done in a vector format easily.

Some examples include:

A simple logo with a photographic background. The logo could be separated from the background and converted to vector, with the background being left as a bitmap.
A picture of a white to grey gradient background with someone standing in front of the background. The background could be converted to vector whilst the foreground is kept as a bitmap, resulting in potentially better compression and image quality.
A bar chart, where the majority of the image would probably benefit from a vector image format more than a bitmap image format, but contains photos within the bar chart or a complex background. The bar chart itself could be converted to vector and the photos and text would be left as bitmaps.
An image of a weather forecast, where there's lots of areas that have plain and/or simple backgrounds, such as the temperature boxes and news rotor at the bottom of the image. Here the areas with simple backgrounds and shapes could again be converted to vector, with the rest being left as a bitmap. It might even be possible to go further and include text recognition and font recognition, but that could be inaccurate and only support a handful of simple fonts (like Arial, Courier New, etc...)
Icons saved as fixed sized bitmaps. These would obviously benefit from being vectors where possible, but some people lack the knowledge of how to make and/or use vector images, or the site they're posting them on does not allow SVG images to be embedded in their forum post.

So, does this sound like a good idea or am I not understanding the fundamentals of image codecs or something? :)

jonsneyers commented 8 years ago

I think you are basically describing PDF :) On Apr 17, 2016 22:13, "PaintNinja" notifications@github.com wrote:

Hello, I've been watching FLIF development as a pro user and been really impressed with the progress being made! :)

I would like to share an idea which may improve compression ratio in some scenarios. I apologise in advance if this sounds like a bad idea, I don't have much knowledge in how image codecs work.

The idea is to mix both vector and bitmap graphics into a single image, allowing for improved compression ratios and possibly image quality in some scenarios. There are often solid shapes or gradients within images where they achieve much better compression as SVGZ files rather than PNG files, but could have more complex scenes in the same image which couldn't be done in a vector format easily.

Some examples include:

-

A simple logo with a photographic background. The logo could be separated from the background and converted to vector, with the background being left as a bitmap.

A picture of a white to grey gradient background with someone standing in front of the background. The background could be converted to vector whilst the foreground is kept as a bitmap, resulting in potentially better compression and image quality.

A bar chart, where the majority of the image would probably benefit from a vector image format more than a bitmap image format, but contains photos within the bar chart or a complex background. The bar chart itself could be converted to vector and the photos and text would be left as bitmaps.

An image of a weather forecast, where there's lots of areas that have plain and/or simple backgrounds, such as the temperature boxes and news rotor at the bottom of the image. Here the areas with simple backgrounds and shapes could again be converted to vector, with the rest being left as a bitmap. It might even be possible to go further and include text recognition and font recognition, but that could be inaccurate and only support a handful of simple fonts (like Arial, Courier New, etc...)

So, does this sound like a good idea or am I not understanding the fundamentals of image codecs or something? :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/FLIF-hub/FLIF/issues/263

PaintNinja commented 8 years ago

Sort of, but PDF is more of a document format than an image format. You can't easily embed PDF files in webpages, you can't open them in normal image viewers, they're not responsive by design like FLIF is, etc...

It would be great if FLIF could not only replace multiple bitmap image formats but also vector formats and for it to be possible to use both in a single image outputted by the FLIF encoder (such as allowing vector graphics to be embedded in a FLIF image), as well as for FLIF to convert parts of inputted bitmap images to vector during encoding.

Say for example we input a 128px x 128px black and white PNG icon of an arrow into the FLIF encoder. The encoder would use an algorithm to detect whether the input image would be more suitable as a vector, and if so convert it to a vector format with an identifier that allows the FLIF decoder to know that it needs to decode the vector parts of the image differently to the bitmap parts of the image. Or, if we input that same icon but from a vector format like SVG, FLIF could use a combination of XWRT and MANIAC compression to achieve better compression ratios than just plain SVG + Gzip.

Or maybe we have a normal photo with a solid white background and someone standing in front of that background. Here, instead of encoding each group of pixels as white for the whole part of the image, you could separate the white background from the foreground and just have a single vector instruction stating to fill the whole background in white. This could potentially result in much better compression than stating which groups of pixels should be white, where it's no longer white because the person standing in front is in the way, then it's white again, etc...

jonsneyers commented 8 years ago

Converting raster to vector automatically is not easy, and I don't think I want to complicate the FLIF encoder to deal with that problem. But what can be done easily is to have a combined SVG+FLIF format, e.g. simply an ar file that contains both a .flif and a .svg.gz, which get overlaid. The FLIF decoder would just decode the FLIF image (which would simply be transparent in the areas which do not contain bitmap parts), standard SVG tools would decode the SVG, and putting the two together is straightforward.

For the "white background" example you don't need vectors, the background would compress very efficiently with just FLIF -- if you would put a semi-transparent foreground on top of a vector white background, it would not be better compressed since the alpha channel would also have to be encoded.

sipa commented 8 years ago

Sounds like DjVu (https://en.wikipedia.org/wiki/DjVu)

PaintNinja commented 8 years ago

Having a combined SVG+FLIF format sounds fantastic! It would allow for people to easily combine vector and bitmap images into a single image on public websites such as image hosting sites or forums, whilst also making it easier for web developers to overlay SVG images on top of FLIF images without needing additional CSS and HTML.

hrj commented 8 years ago

SVG allows referring to images (and also embedding images via data-URI). So even now, in contexts where SVG and FLIF is supported, one could combine vector + raster images. See this Q&A for example.

hrj commented 8 years ago

Though, note that data-URIs are very inefficient (Base64 encoding). Would have been nicer if raw binary attachments could be added to SVG.

AVHon commented 8 years ago

Base64 encoded images in SVG can be compacted by using SVGZ, which is just a gzipped SVG file. Support for SVGZ isn't quite as common as support for SVG; you can see if your browser supports it by loading this image. SVGZ files with Base64 images embedded inside should be nearly the same size as - or smaller than - separate SVG and bitmap files.

I think that this idea is outside the scope of the FLIF format, and suggest that the issue be closed.

jonsneyers commented 8 years ago

Closing.

FLIF-hub / FLIF

Idea: Vector and bitmap format mixing #263

A simple logo with a photographic background. The logo could be separated from the background and converted to vector, with the background being left as a bitmap.

A picture of a white to grey gradient background with someone standing in front of the background. The background could be converted to vector whilst the foreground is kept as a bitmap, resulting in potentially better compression and image quality.

A bar chart, where the majority of the image would probably benefit from a vector image format more than a bitmap image format, but contains photos within the bar chart or a complex background. The bar chart itself could be converted to vector and the photos and text would be left as bitmaps.