Processing animated GIFs while keeping transparency

karolyi commented 4 years ago

Hey,

after a week of fiddling around with Pillow and reading/testing the source, I've written a module that will allow for processing an animated GIF while retaining its transparency throughout all frames.

In my case, I needed to create thumbnails (various sizes) and watermark them, then bumped into the problem of Pillow not being able to properly handle these files.

Attached you will find a pillowtest.zip that contains the code and demo files for the code.

For simplicity's sake, here's the source:

from pathlib import Path
from typing import Tuple, Iterable
from collections import defaultdict
from operator import itemgetter
from random import randrange
from itertools import chain

from PIL.Image import Image
from PIL.Image import open as image_open
from PIL.ImageSequence import Iterator as SeqIterator

with image_open(fp=Path(__file__, '..', 'watermark.png')) as image:
    im_wm = image  # type: Image
    im_wm.load()
im_walk = image_open(fp=Path(__file__, '..', 'walk.gif'))  # type: Image
im_anim = image_open(fp=Path(__file__, '..', 'animated.gif'))  # type: Image
im_trans = image_open(fp=Path(__file__, '..', 'transparent.gif'))  # type: Image
im_murica = image_open(fp=Path(__file__, '..', 'murica.gif'))  # type: Image
with image_open(fp=Path(__file__, '..', 'test.jpg')) as image:
    im_jpg = image  # type: Image
    im_jpg.load()
PALETTE_SLOTSET = set(range(256))

class TransparentAnimatedGifConverter(object):
    _PALETTE_SLOTSET = set(range(256))

    def __init__(self, img_rgba: Image, alpha_threshold: int = 0):
        self._img_rgba = img_rgba
        self._alpha_threshold = alpha_threshold

    def _process_pixels(self):
        'Set the transparent pixels to the color 0.'
        self._transparent_pixels = set(
            idx for idx, alpha in enumerate(
                self._img_rgba.getchannel(channel='A').getdata())
            if alpha <= self._alpha_threshold)

    def _set_parsed_palette(self) -> list:
        'Parse the RGB palette color `tuple`s from the palette.'
        palette = self._img_p.getpalette()
        self._img_p_used_palette_idxs = set(
            idx for pal_idx, idx in enumerate(self._img_p_data)
            if pal_idx not in self._transparent_pixels)
        self._img_p_parsedpalette = dict(
            (idx, tuple(palette[idx * 3:idx * 3 + 3]))
            for idx in self._img_p_used_palette_idxs)

    def _get_similar_color_idx(self, translated_set: set):
        'Return a palette index with the closest similar color.'
        old_color = self._img_p_parsedpalette[0]
        dict_distance = defaultdict(list)
        for idx in range(1, 256):
            color_item = self._img_p_parsedpalette[idx]
            if color_item == old_color:
                return idx
            distance = sum((
                abs(old_color[0] - color_item[0]),  # Red
                abs(old_color[1] - color_item[1]),  # Green
                abs(old_color[2] - color_item[2])))  # Blue
            dict_distance[distance].append(idx)
        return dict_distance[sorted(dict_distance)[0]][0]

    def _remap_palette_idx_zero(self):
        'Since the first color is used in the palette, remap it.'
        free_slots = self._PALETTE_SLOTSET - self._img_p_used_palette_idxs
        new_idx = free_slots.pop() if free_slots else \
            self._get_similar_color_idx()
        self._img_p_used_palette_idxs.add(new_idx)
        self._palette_replaces['idx_from'].append(0)
        self._palette_replaces['idx_to'].append(new_idx)
        self._img_p_parsedpalette[new_idx] = self._img_p_parsedpalette[0]
        del(self._img_p_parsedpalette[0])

    def _get_unused_color(self) -> tuple:
        """
        Return a color for the palette that does not collide with any
        other already in the palette.
        """
        used_colors = set(self._img_p_parsedpalette.values())
        while True:
            new_color = (randrange(256), randrange(256), randrange(256))
            if new_color not in used_colors:
                return new_color

    def _process_palette(self):
        """
        Adjust palette to have the zeroth color set as transparent.
        Basically, get another palette index for the zeroth color.
        """
        self._set_parsed_palette()
        if 0 in self._img_p_used_palette_idxs:
            self._remap_palette_idx_zero()
        self._img_p_parsedpalette[0] = self._get_unused_color()

    def _adjust_pixels(self):
        'Convert the pixels into theire new values.'
        if self._palette_replaces['idx_from']:
            trans_table = bytearray.maketrans(
                bytes(self._palette_replaces['idx_from']),
                bytes(self._palette_replaces['idx_to']))
            self._img_p_data = self._img_p_data.translate(trans_table)
        for idx_pixel in self._transparent_pixels:
            self._img_p_data[idx_pixel] = 0
        self._img_p.frombytes(data=bytes(self._img_p_data))

    def _adjust_palette(self):
        'Modify the palette in the new `Image`.'
        unused_color = self._get_unused_color()
        final_palette = chain.from_iterable(
            self._img_p_parsedpalette.get(x, unused_color) for x in range(256))
        self._img_p.putpalette(data=final_palette)

    def process(self) -> Image:
        'Return the processed mode `P` `Image`.'
        self._img_p = self._img_rgba.convert(mode='P')
        self._img_p_data = bytearray(self._img_p.tobytes())
        self._palette_replaces = dict(idx_from=list(), idx_to=list())
        self._process_pixels()
        self._process_palette()
        self._adjust_pixels()
        self._adjust_palette()
        self._img_p.info['transparency'] = 0
        self._img_p.info['background'] = 0
        return self._img_p

def _create_animated_gif(image: Image, size: tuple) -> Tuple[Image, dict]:
    'If the image is a GIF, create an its thumbnail here.'
    save_kwargs = dict()
    wm = im_wm.convert(mode='RGBA')

    def _thumbnails() -> Image:
        'Inner iterator for frames.'
        for idx, frame in enumerate(frames):  # type: Image
            thumbnail = frame.copy()  # type: Image
            # _print_transparent_count(thumbnail)
            thumbnail_rgba = thumbnail.convert(mode='RGBA')
            # print(list(thumbnail.getchannel(channel='A').getdata()))
            thumbnail_rgba.thumbnail(size=frame.size, reducing_gap=3.0)
            thumbnail_rgba.paste(im=wm, box=(frame.tell(), frame.tell()), mask=wm)
            converter = TransparentAnimatedGifConverter(img_rgba=thumbnail_rgba)
            thumbnail_p = converter.process()  # type: Image
            # print(frame.info, thumbnail_p.info, frame.size, thumbnail_p.size)
            yield thumbnail_p

    frames = SeqIterator(im=image)
    output_image = next(_thumbnails())
    save_kwargs.update(
        format='GIF',
        save_all=True,
        optimize=False,
        append_images=list(_thumbnails()),
        disposal=2)  # Other disposals don't work
    return output_image, save_kwargs

def run(*args):
    outfilepath = Path(__file__).parent.joinpath('out.gif')
    # im = im_trans
    # im = im_anim
    # im = im_walk
    im = im_murica
    output_image, save_kwargs = _create_animated_gif(image=im, size=im.size)
    output_image.save(fp=outfilepath, **save_kwargs)
    # from IPython import embed; embed()

if __name__ == '__main__':
    run()

Do what you want with it, I don't care. I have tried to touch the source but to me it seems such a huge mess that I rather didn't touch it after discovering/debugging the code execution paths it. You can test the conversion of all the images within the zip, most of which is a transparent animated GIF, works with every one of them. The key thing is, transparent palette indexes should be the same throughout all frames, that's what this module does.

Things I wanted to do but it seems impossible without a huge rewrite:

Palette optimization is a thing. With bytes.translate (see the code) one would be able to process and shrink palettes very efficiently, but it makes no sense to do it in here because the final palette will get overridden in GifImagePlugin.py anyways.
Compression on the individual frames: at current time, Pillow creates uncompressed GIFs out of compressed gifs, and the save logic is in an external C module. I'm not a C coder, but I wold really welcome if someone would be able to do the LZW compression there.

All in all, the gif module in my opinion is in for a huge rewrite, but I can't offer my help since I'm busy with something else right now. Also, the compression in the C extension is highly needed to cut the generated file sizes less then half, most of the time.

karolyi commented 4 years ago

Here's an image where the decoding (image.convert(mode='RGBA')) screws up the palette for some reason: I've tried it out with the test code I posted above, it seems the RGBA conversion is the culprit, which is not my table.

karolyi commented 4 years ago

This one's even worse, the culprit again is the RGBA conversion.

radarhere commented 4 years ago

Please refrain from posting inappropriate images.

karolyi commented 4 years ago

@radarhere there was no nudity on either of them, you're being a bit too sensitive. You deleted images that can be used as test for decoding failures. Fine with me though, as it seems by my experience that these images are around 2% of the animgifs I see wrongly decoded. But still, something's off with them.

Good to see they're still available in the history.

Dragorn421 commented 4 years ago

With gifs from my issue as you asked, it has the same issues as Pillow (black borders and bad frame combine), since it doesn't do anything different for loading gifs. It also looses information on frame duration (makes them all the same).

outpeepocreepo

outpepepls

karolyi commented 4 years ago

I've checked your images and seen that during the conversion, the duration info values come through. So the culprit for F-ing up the durations is the GIF saver logic, seemingly.

Septem151 commented 4 years ago

Despite @karolyi 's slightly inappropriate picture, this is exactly the results I'm getting as well and it's extremely frustrating. The first frame appears fine, yet no matter what I try to do the frames after the first always look that way. Is there any workaround for this at all?

egocarib commented 4 years ago

@karolyi thank you for the workaround code. I also suffered for a few days with black pixels replacing my transparency. Your code worked to correct the issue for me with some slight adjustments to meet my need.

The one limitation I have observed is that the code above does not play well with partial alpha transparency. It seems to "flatten" the alpha to fully solid colors. But it works well for transparent, moving gif animations that use only alpha channels 0 and 255.

If it's helpful to anyone else, here is a slightly adapted version of karolyi's code that is a bit easier to plug-and-play with. You can simply call the save_transparent_gif method of this code with your list of images and durations. Minimal (if any) editing should be required:

https://gist.github.com/egocarib/ea022799cca8a102d14c54a22c45efe0

I hope that this issue can eventually be fixed.

StupidCrow-yuan commented 3 years ago

With gifs from my issue as you asked, it has the same issues as Pillow (black borders and bad frame combine), since it doesn't do anything different for loading gifs. It also looses information on frame duration (makes them all the same).

Did you have fixed the bug?

StupidCrow-yuan commented 3 years ago

hanbao