search

SixLabors / ImageSharp

:camera: A modern, cross-platform, 2D Graphics library for .NET
https://sixlabors.com/products/imagesharp/
Other
7.34k stars 847 forks source link

Add smart animated format conversion. #2588

Closed JimBobSquarePants closed 9 months ago

JimBobSquarePants commented 9 months ago

Prerequisites

Description

Note: Most of the file changes here are new references images that were generated to cater for changes in the color distance calculator that improved lookup accuracy.

This PR adds some internal tooling that allows us to convert between animated image formats without the user having to specify the mechanics for each frame to ensure converting between formats returns an equivalent result.

For v4 I plan to update the metadata formats to introduce public bridging types and methods to create smart defaults when switching formats but for now I had to use an internal, non-extensible approach.

For v3.1 we will support bi-directional conversion between gif, png, and webp formats. Full color table mapping is performed between formats when present as well as duration, and disposal, blend modes where applicable.

Here's an example of an automatic conversion.

Gif (45.2KB)

Encode_AnimatedFormatTransform_FromGif_Rgba32_giphy

Png (33.3KB)

Encode_AnimatedFormatTransform_FromGif_Rgba32_giphy

In addition to the conversion functionality all 3 animated formats have been enhanced with shared methodology that allows the optimization of the frames to only include changed data. This prevents the potential explosion in file size (multi MBs) that can be caused when re-encoding the frames that are coalesced during decode.

This example from gifiddle demonstrates the active encoded area of a frame following that optimization.

image

For many gifs this actually leads to an improvement over the incoming file size. For example, the original of the gif above is 52.3KB

JimBobSquarePants commented 9 months ago

@antonfirsov @tocsoft I need your help if you have time?

I'm seeing allocation cleanup failure reports on Linux/Mac and I can recreate them on my old MacBook. Thing is... I've touched nothing related to memory management nor does this test touch code I've changed with my PR.

Is there any way to easily diagnose the cause? I tried using dotmemory Unit in Rider but it won't give the option to run the test in the manner that allows profiling memory.

Running this test in isolation is enough to cause failure. [xUnit.net 00:00:27.79] TiffDecoder_CanDecode_JpegCompressed(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32]) [FAIL]

EDIT.... OK. It was failing locally but now it suddenly isn't. Could it be a memory pressure thing triggered by my additional tests?

EDIT... It was ME!!

stefannikolei commented 9 months ago

Forget this...

I removed my local copy of the repo and did a clean checkout. Now everything works on the main branch.. strange

@antonfirsov @tocsoft I need your help if you have time?

I'm seeing allocation cleanup failure reports on Linux/Mac and I can recreate them on my old MacBook. Thing is... I've touched nothing related to memory management nor does this test touch code I've changed with my PR.

Is there any way to easily diagnose the cause? I tried using dotmemory Unit in Rider but it won't give the option to run the test in the manner that allows profiling memory.

Running this test in isolation is enough to cause failure. [xUnit.net 00:00:27.79] TiffDecoder_CanDecode_JpegCompressed(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32]) [FAIL]

EDIT.... OK. It was failing locally but now it suddenly isn't. Could it be a memory pressure thing triggered by my additional tests?

I tried running main first on my Mac. But not even there all tests go green.

     Encode_AnimatedLossy<Rgba32>(provider: Webp/leo_animated_lossy.webp[Rgba32])

SixLabors.ImageSharp.ImageFormatException Read unexpected webp chunk data at SixLabors.ImageSharp.Formats.Webp.WebpThrowHelper.ThrowImageFormatException(String errorMessage) in /Users/stefannikolei/projects/github/sixlabors/ImageSharp/src/ImageSharp/Formats/Webp/WebpThrowHelper.cs:line 14

   CanLimitHwIntrinsicFeaturesWithIntrinsicsParam
   CanLimitHwIntrinsicFeaturesWithSerializableAndIntrinsicsParams
   CanLimitHwIntrinsicFeaturesWithSerializableParam

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException Remote process failed with an unhandled exception. Child exception: Xunit.Sdk.FalseException: Assert.False() Failure Expected: False Actual: True

stefannikolei commented 9 months ago

following tests are failing on my machine

     GifDecoderTests
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/cheers.gif[Rgba32], expectedFrameCount: 93)
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/issues/issue403_baddescriptorwidth.gif[Rgba32], expectedFrameCount: 36)
     Issue1668_InvalidColorIndex<Rgba32>(provider: Gif/issues/issue1668_invalidcolorindex.gif[Rgba32])
   PngEncoderTests
     Encode_APng<Rgba32>(provider: Png/animated/apng.png[Rgba32])     
    TiffDecoderTests
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_deflate_withPreview.tiff[Rgba32])
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_lzw.tiff[Rgba32])
     TiffDecoder_CanDecode_JpegCompressed<Rgba32>(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32])

I tried debugging it.

In my eyes (looking at the failing png test) The problem starts to manifest here:

image

In all other iterations the width was 32 and resulted in bytesPerFrameScanLine = 129. Here the width is 10

A naive fix (the unit test passes after this:

image
JimBobSquarePants commented 9 months ago

following tests are failing on my machine

     GifDecoderTests
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/cheers.gif[Rgba32], expectedFrameCount: 93)
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/issues/issue403_baddescriptorwidth.gif[Rgba32], expectedFrameCount: 36)
     Issue1668_InvalidColorIndex<Rgba32>(provider: Gif/issues/issue1668_invalidcolorindex.gif[Rgba32])
   PngEncoderTests
     Encode_APng<Rgba32>(provider: Png/animated/apng.png[Rgba32])     
    TiffDecoderTests
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_deflate_withPreview.tiff[Rgba32])
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_lzw.tiff[Rgba32])
     TiffDecoder_CanDecode_JpegCompressed<Rgba32>(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32])

I tried debugging it.

In my eyes (looking at the failing png test) The problem starts to manifest here:

image

In all other iterations the width was 32 and resulted in bytesPerFrameScanLine = 129. Here the width is 10

A naive fix (the unit test passes after this:

image

Well spotted! Yes, that's definitely a bug. The interlaced version of the method was doing the correct thing.