Support animated GIFs with transparency as RGBA frames with frame delay

[Boscop]

For a project of mine I was looking for a crate that could load animated GIFs which I wanted to render with glium. Since this crate didn't have support for loading animated GIFs (I was surprised, because animation is the main reason why GIFs are used) I wrote my own loader based on this crate. It would be great to integrate animated GIF support into this crate, so that other projects can benefit from it as well. This is my animated GIF loader / renderer code:

pub struct AnimGif<'a> {
    texture: glium::Texture2d,
    vertex_buffer: glium::vertex::VertexBufferAny,
    index_buffer: glium::index::IndexBufferAny,
    program: glium::Program,
    params: glium::DrawParameters<'a>,
    frame_count: u32,
    height: u32,
}
impl<'a> AnimGif<'a> {
    pub fn new<F: Facade, P: AsRef<Path>>(display: &F, path: P) -> AnimGif<'a> {
        use gif;
        use gif::SetParameter;
        use std::fs::File;
        let mut decoder = gif::Decoder::new(File::open(path).unwrap());
        decoder.set(gif::ColorOutput::RGBA);
        let mut decoder = decoder.read_info().unwrap();
        let size = (decoder.width() as u32, decoder.height() as u32);
        let frame_size = (size.0 * size.1 * 4) as usize;
        let mut gif_data = Vec::<u8>::new();
        let mut frame_count: u32 = 0;
        while let Some(frame) = decoder.read_next_frame().unwrap() {
            assert_eq!(frame.delay, 10);
            use image::GenericImage;
            let cur_frame = vec![0u8; frame_size];
            let src = image::ImageBuffer::<image::Rgba<u8>, Vec<u8>>::from_raw(frame.width as u32, frame.height as u32, frame.buffer.clone().into_owned()).unwrap();
            let mut dst = image::ImageBuffer::<image::Rgba<u8>, Vec<u8>>::from_raw(size.0, size.1, cur_frame).unwrap();
            dst.copy_from(&src, frame.left as u32, frame.top as u32);
            gif_data.extend(dst.into_raw());
            frame_count += 1;
        }
        let image = glium::texture::RawImage2d::from_raw_rgba_reversed(gif_data, (size.0, size.1 * frame_count));
        let texture = glium::texture::Texture2d::new(display, image).unwrap();

        #[derive(Copy, Clone)]
        struct Vertex {
            position: [f32; 2],
            tex_coords: [f32; 2],
        }

        implement_vertex!(Vertex, position, tex_coords);

        let shape = vec![
            Vertex { position: [0.0, -1.0], tex_coords: [0.0, 0.0] },
            Vertex { position: [0.0,  0.0], tex_coords: [0.0, 1.0] },
            Vertex { position: [1.0,  0.0], tex_coords: [1.0, 1.0] },
            Vertex { position: [1.0, -1.0], tex_coords: [1.0, 0.0] },
        ];

        let vertex_buffer = glium::VertexBuffer::new(display, &shape).unwrap().into_vertex_buffer_any();

        let index_buffer = glium::index::IndexBuffer::new(display, glium::index::PrimitiveType::TrianglesList, &[0u16,1,2,0,2,3]).unwrap().into();

        let vertex_shader_src = r#"
            #version 410

            layout(location = 0) in vec2 position;

            in vec2 tex_coords;
            out vec2 v_tex_coords;

            uniform mat4 matrix;

            void main() {
                v_tex_coords = tex_coords;
                gl_Position = matrix * vec4(position, 0.0, 1.0);
            }
        "#;

        let fragment_shader_src = r#"
            #version 430
            buffer layout(packed);

            in vec2 v_tex_coords;

            uniform sampler2D tex;
            uniform float cur_frame_offset;
            uniform float frame_count;
            uniform float transparency;

            out vec4 color;

            void main() {
                color = texture(tex, vec2(v_tex_coords.x, (v_tex_coords.y - cur_frame_offset) / frame_count));
                color.a *= transparency;
            }
        "#;

        let program = glium::Program::from_source(display, vertex_shader_src, fragment_shader_src, None).unwrap();

        let params = glium::DrawParameters {
            blend: glium::Blend::alpha_blending(),
            .. Default::default()
        };

        AnimGif {
            texture: texture,
            vertex_buffer: vertex_buffer,
            index_buffer: index_buffer,
            program: program,
            params: params,
            frame_count: frame_count,
            height: size.1,
        }
    }
    pub fn draw<T: Surface>(&mut self, target: &mut T, pos: Vector2<f64>, mut size: Vector2<f64>, angle: f32, transparency: f32, time: f32) {
        let resolution = target.get_dimensions();
        let resolution = (resolution.0 as f32, resolution.1 as f32);

        let (w, h) = target.get_dimensions();
        let (w, h) = (w as f32, h as f32);
        let tx = (pos.x as f32 / w - 0.5) * 2.0;
        let ty = (1.0 - pos.y as f32 / h - 0.5) * 2.0;
        if size == Vector2::new(0., 0.) {
            size = Vector2::new(self.texture.get_width() as f64, self.height as f64);
        }
        let sx = size.x as f32 / w * 2.0;
        let sy = size.y as f32 / h * 2.0;

        let tr_to_center = Matrix4::from_translation(Vector3::new(-0.5, 0.5, 0.));
        let tr_to_topleft = tr_to_center.invert().unwrap();
        let translation = Matrix4::from_translation(Vector3::new(tx, ty, 0.));
        let scale = Matrix4::from_nonuniform_scale(sx, sy, 1.);
        let rotation = Matrix4::from(Quaternion::from_angle_z(Rad::new(angle)));

        let matrix = translation * scale * tr_to_topleft * rotation * tr_to_center;

        let uniforms = uniform! {
            matrix: array4x4(matrix),
            tex: &self.texture,
            cur_frame_offset: (time / 0.100 /* frame time */).floor(),
            frame_count: self.frame_count as f32,
            transparency: transparency,
        };
        target.draw(&self.vertex_buffer, &self.index_buffer, &self.program, &uniforms, &self.params).unwrap();
    }
    pub fn size(&self) -> Vector2<f64> {
        Vector2::new(self.texture.get_width() as f64, self.height as f64)
    }
}

It iterates over all the frames and expands them to the same size, then concatenates them into one vertical texture atlas. cur_frame_offset is used to index that texture during rendering to get the current frame of the animation. As you can see it assumes that the frame delay is always 10 because I was only dealing with those kind of GIFs but it could easily be generalized. The frame delay would have to be stored for each frame, and accumulated and then used for the cur_frame_offset uniform during rendering. Also there are potentials for optimizations/preallocations.. But this is just a start to get the discussion going.

The animated GIF support in this crate shouldn't be specific to glium / opengl but should be designed with that in mind, so that rendering the animated GIFs as textures is easy.

Also it would be useful to be able to save an image sequence as an animated GIF with given frame delay.

What do you think?

[nwin]

As you noticed, this library does in fact support the loading of animated images since all frames are accessible. Furthermore it also allows to output animated images (with delay) as it is illustrated in the example code in the readme file.

Unfortunately the support is quite rough as it did not have any users. I'm very interested to get input for this topic. What exactly are you missing, what should be added?

Imho storing information about the current frames' timely offset doesn't make much sense because this information is deeply tied to the implementation of the presenter. If you would imagine an implementation which just sleeps for delay ms you would not need this information.

[nwin]

Does #33 cover your intention?

[Boscop]

Right, it does support it but people who don't know about the specifics of the GIF format (e. g. that frames can have different sizes) and just want to load animated GIFs as an image sequence will take a long time to figure out how to load them like this. I think the code that loads all the frames as RGBA image buffers of the same size will be part of every animated GIF loading system in user code so it would make sense to include this functionality in this crate.

re: frame delay, I meant that the output shouldn't just contain a Vec of image buffers but also each frame's relative delay to the frame before (so the user code can choose to integrate the frame delays or sleep etc.)

Also to be able to save a sequence of image buffers (with associated frame delays) as animated GIF would be useful. (The inverse operation of the loading, where each frame is cropped to the bounding box of its non transparent pixels).

[nwin]

I totally agree to you on the image buffer thing. I’m currently reworking the existing image buffer from image in order to move it into it’s own crate such that it can be used in the decoder libs as well. I’m currently a bit stuck on how to handle color models properly.

Since the delay specifies the time until the next image is displayed I do not see how storing the time to the previous image would help.

I am still not convinced how equally sized buffers would always help, you could for example just blit the next frame onto the existing image/background color and you’re fine. On the other hand, the fact that your code ignores the disposal method convinces me that there should be a convenience method in addition to the "low level" interface.

Unfortunately it is not beneficial to implement the iterator trait such that a Vec of images can be provided via Iterator::collect since iterators cannot fail (i.e. use Result).

[kosinix]

I think this issue can simply be solve by expanding the documentation.

@nwin

Furthermore it also allows to output animated images (with delay) as it is illustrated in the example code in the readme file.

Unfortunately, the doc does not give an example on how to make a multi frame gif with delays and disposal set for each frame. Maybe this is where the confusion of users stem from.

The from_rgba/from_rgb APIs are already good high level APIs for creating animated GIF with custom delays for each frame. It just needs an example code.

[Boscop]

Thanks, I didn't even know I have to care about the disposal method. I thought the frames would already be decoded to account for it... Looking into it, it seems like it's easy to get the disposal wrong, (Netscape got it wrong). So it would be useful to have an interface so that client code doesn't have to implement that but can just get the frames such that when they are rendered on a cleared background they look like the GIF.

@kosinix: Yeah, maybe an example that takes the disposal method into account would be sufficient. And this example would have a function that could be factored out of the example, into the gif crate, that applies a frame to another frame with a given disposal method. E.g. as a method apply(old, new) of gif::DisposalMethod.

[nwin]

@boscop Unfortunately nobody gets animated gifs right. Frames with a delay of zero are rendered incorrect in all browsers I know of.

[kornelski]

I'm writing GIF converter and for it I need both raw frames with palettes, as well as RGBA rendering of all of it together.

Some sort of abstraction for gif's concept of "screen" would be very useful:

while let Some(frame) = reader.read_next_frame()? {
    screen.blit(frame);
    screen.rgba(); // -> &[RGBA]
}

[kornelski]

In case anybody needs it, I've implemented it:

edit: now it's a crate! https://lib.rs/crates/gif-dispose

[nwin]

Thank you, but the third to last line is the reason why it is still is as it is. It is really not trivial to create an ergonomic but safe (i.e. do not use unsafe for such “trivial” things) abstraction.

[kornelski]

Oh, that unsafe line was just a microoptimization. It was quite unnecessary, so I've removed it. Now it's 100% safe pure Rust.

[nwin]

That is not what I meant. It get’s quite tricky if you only want to define a shim over the underlying slice of primitives. Meaning not copying the data, which is what your code did before.

Something like color_model/mod.rs#L54 is really difficult to avoid, because you cannot write generic code that abstracts over ownership, mutability or integer numbers.

[kornelski]

Sorry, I completely don't understand what you're saying.

• Previous frame disposal mode requires copying. I don't see any reasonable way around it, because by design of GIF, the screen is an accumulated mutable state. There are some optimizations possible (e.g. save and restore only frame area not the whole screen, maybe resize previous vec instead of reallocating), but in general it has to copy.

• the frame has palette as [u8]. It could be used as-is by doing bytewise copy or R, G, B components. I only copy it for my convenience, and the copy is insignificantly tiny.

• If you mean allowing arbitrary Pixel as backing store instead of RGBA, that should be possible.

[nwin]

All I'm saying is, that I'm completely with you but it takes time to implement because I it is harder to create a wrapper than i thought.

[kornelski]

Is there something I can help with?

I think the disposal code should live in the gif crate. What are your requirements for including it?

[Boscop]

Any updates on this?