Faster GIF encoding

[mattdesl]

I notice the GIF encoder is pretty slow and hangs, and also seems to be a two-pass approach. You might like to look into gifenc which should be faster. Just to note, it uses a different quantization algorithm, so it may produce different results.

You can see an example in action below, rendering 150 frames at 1024x768px renders in ~1.5 seconds, compared to your tool which is something like 30+ seconds. :smile:

https://looom-tools.netlify.app/ (click the circle/record button)

It requires a little more setup but can give you some additional flexibility in terms of quantization, worker structure, and dithering.

Two example implementations:

• looom-tools (uses a worker, some extra features)
  • Client: https://github.com/mattdesl/looom-tools/blob/a9f455eeab3e43af6775e903cfafe6e9568dea4d/site/components/record.js#L250
  • Worker: https://github.com/mattdesl/looom-tools/blob/a9f455eeab3e43af6775e903cfafe6e9568dea4d/site/components/gifworker.js
• artblocks-renderer (simpler but includes dithering option)
  • Client: https://github.com/mattdesl/artblocks-renderer/blob/24b3c64fd8472d003d5547723f2ff9f528fc4378/src/recording.js#L140

There is also room for a two-pass approach with gifenc, such as generating a single palette from all frames in the first pass, and in the second pass doing the encoding with that fixed palette. This can produce better consistency across frames but is slower (although, to be honest, probably still faster than your current encoder).

Also pinging @jesi-rgb who is doing similar work with p5js, it would be nice to consolidate some of these ideas around fast GIF encoding for creative coding tools. See here and here.

[jesi-rgb]

Thanks so much for pinging, @mattdesl! It really is an honor.

The state of the PR for adding native gif encoding is very much not finished, but enough so to comply with the GSoC standards.

State of the art

There are still a ton of features I'd love to add such as dithering or better UX experience in general. Right now, I'd say that creative experts would use it for quick draft renders but will probably look otherwise (FFMPEG) for high quality, publishable work. I'll try to improve it as I learn, since I'm very much a newbie at this point :P

Implementation

As for the 2-pass approach, I indeed implemented that on what will become the new saveGif() function in p5.js, which is currently not public for everyone to use, but will be. And I take the approach that was explained in the blog, as you referenced!

Benchmarking

The details are:

• 858 square canvas
• Using saveGif
  • Saving for 2 seconds at 60 fps for a total of 120 frames.
• Somewhat busy sketch (see below)
• Seeded for every run and browser
• M1 Mac Mini with 16GB RAM.

Results

• Chrome:
  • Run 1: 8.40 seconds
  • Run 2: 8.60 seconds
  • Run 3: 8.50 seconds
• Firefox:
  • Run 1: 19.38 seconds (lol)
  • Run 2: 19.40 seconds
  • Run 3: 19.55 seconds
• Safari
  • Run 1: 14.19 seconds (lol again)
  • Run 2: 13.93 seconds
  • Run 3: 14.21 seconds

These results are not to be taken too seriously at all as I just run the algorithm and measured the time with my phone's clock app. But the fact that Firefox < Safari < Chrome is always consistent.

Conclusion

If I can help or can do anything at all to help on the release of the tool, I will be more than happy to contribute. Just bear in mind that I'm still figuring my way around all this technologies :)

The deadline for the GSoC is Sept 12th. By that time, the PR will have been merged and this feature will be available in the next release of p5.js, about which I have no idea of the scheduling.

That's it!

[raphaelameaume]

Indeed, the current implementation of GIF encoding is quite slow comparing to other encoders at your disposal in Fragment. To be honest, I added it using gif.js but I haven't use it much because of its slowness and my (total) absence of knowledge regarding the topic. This article by @jesi-rgb is a pure gem! And this transparency optimization trick is quite mind-blowing.

I missed on gifenc, which looks pretty cool. I think one of the first things we could do would be to switch from gif.js to gifenc in its current state.

Regarding the 2 passes, I think having the implementation possible would be a nice addition as result might really differ depending on what you'd be working on.

@jesi-rgb considering you implemented that on saveGif(), do you think this should live as a core function of the gif encoder or at a higher-level ?

@mattdesl the faster the better so using workers would be great. Curious about the numbers of workers that should be used in this case ? You're comparing without and with 4 workers in the gifenc README file but you seem to be using only 2 in the loom-tools implementation. Isn't it more workers, the faster the encoding happens ?

I think if we end up having two implementations, we could add a in fragment exports module only when gif is selected, letting the user decides (but we can point to some helpful documentation regarding the pros and cons of each technique).

You both have a way better understanding of the situation here, so please correct me if my questions don't make sense 😄

jesi-rgb commented 2 years ago

This article by @jesi-rgb is a pure gem! And this transparency optimization trick is quite mind-blowing.

Thanks so much, Raphaël <3

---

@jesi-rgb considering you implemented that on saveGif(), do you think this should live as a core function of the gif encoder or at a higher-level?

I am not aware of the architecture of the application, and would have to dive a bit deeper into it. I am glad to hop on a quick call so you can explain where things are more or less and have a better understanding of the tool's structure, if that's okay with you and have time for it. No need to rush, but I like to meet people in "person" :P

In any of the cases, given that I already implemented both the encoding of the frames in p5.js and the transparency optimization, it'd be a shame not to do it here. I would be very glad to contribute to that.

raphaelameaume commented 2 years ago

Not saying it should be implemented in Fragment, but maybe in gifenc or somewhere in between. Sharing the point of view of @mattdesl here:

it would be nice to consolidate some of these ideas around fast GIF encoding for creative coding tools.

I don't think your work should be tied to Fragment as it seems it could benefit other creative tools. Fragment is just wrapping the different encoders already out there, see here for details: https://github.com/raphaelameaume/fragment/blob/main/src/client/app/lib/canvas-recorder/GIFRecorder.js

jesi-rgb commented 2 years ago

Aha! So sorry, misunderstood the message.

jesi-rgb commented 2 years ago

Been putting some thoughts on the question these days and I believe that the functionality we are talking about could be implemented in gifenc, but really belongs outside of it. The job of a gif encoder is just that. How you take the frames and what you do with them belongs elsewhere.

I could try my best as a gifenc wrapper, a small tool on top of gifenc that takes as input a series of frames and outputs the GIF, with already preset options for ease of use. Does that make any sense?

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[jesi-rgb]

This article by @jesi-rgb is a pure gem! And this transparency optimization trick is quite mind-blowing.

Thanks so much, Raphaël <3

---

@jesi-rgb considering you implemented that on saveGif(), do you think this should live as a core function of the gif encoder or at a higher-level?

I am not aware of the architecture of the application, and would have to dive a bit deeper into it. I am glad to hop on a quick call so you can explain where things are more or less and have a better understanding of the tool's structure, if that's okay with you and have time for it. No need to rush, but I like to meet people in "person" :P

In any of the cases, given that I already implemented both the encoding of the frames in p5.js and the transparency optimization, it'd be a shame not to do it here. I would be very glad to contribute to that.

[raphaelameaume]

Not saying it should be implemented in Fragment, but maybe in gifenc or somewhere in between. Sharing the point of view of @mattdesl here:

it would be nice to consolidate some of these ideas around fast GIF encoding for creative coding tools.

I don't think your work should be tied to Fragment as it seems it could benefit other creative tools. Fragment is just wrapping the different encoders already out there, see here for details: https://github.com/raphaelameaume/fragment/blob/main/src/client/app/lib/canvas-recorder/GIFRecorder.js

[jesi-rgb]

Aha! So sorry, misunderstood the message.

[jesi-rgb]

Been putting some thoughts on the question these days and I believe that the functionality we are talking about could be implemented in gifenc, but really belongs outside of it. The job of a gif encoder is just that. How you take the frames and what you do with them belongs elsewhere.

I could try my best as a gifenc wrapper, a small tool on top of gifenc that takes as input a series of frames and outputs the GIF, with already preset options for ease of use. Does that make any sense?