search

wader / static-ffmpeg

Multi-arch docker image with ffmpeg/ffprobe binaries built as hardened static PIE binaries with no external dependencies
https://hub.docker.com/r/mwader/static-ffmpeg/
MIT License
270 stars 65 forks source link

Caching research #252

Closed binoculars closed 2 years ago

binoculars commented 2 years ago

After some tweaks and enabling caching, I was able to get the build down to 10 minutes for an added library (libsnappy) and 1 minute for non-Dockerfile changes. Updating x265 took about 17 minutes. I think this is the way to go for speed.

What I did to my repo:

wader commented 2 years ago

After some tweaks and enabling caching, I was able to get the build down to 10 minutes for an added library (libsnappy) and 1 minute for non-Dockerfile changes. Updating x265 took about 17 minutes. I think this is the way to go for speed.

Interesting, thanks for trying things out!

What I did to my repo:

  • Move most ARGs to the top

It actually used to be like that but at some point i decided it was nice to not spread things out if possible.

  • Inherit ARGs only where needed
  • Enable caching for GitHub actions docker builds

This i should really do. Do you know what limits there are on cache sizes? some expire time etc? thinking if there are some limits that would make caching not as useful in practice.

  • Parallelize the builds by breaking them up into multi-stage

Is nice but as i mentioned earlier not sure yet what i think about the increased complexity of the Dockerfile.

  • Use COPY command to copy only files needed from previous stages
  • Added several new libraries

Haven't used ladspa and frei0r myself, they seem to use .so files? do you know if building with musl libc could make this a problem? are glibc built plugins compatible?

Noticed libcaca was removed? i remember there was some x11 issue etc?

libvo-amrwbenc and snappy would be nice to add

About installing alpine package or build from source: I've chosen to built from source if the library has updates for features or security, especially decoders, that would take time to end up in alpine. Otherwise also if there are some other issue lik drags in deps we don't want or not built with optimisation/hardening. But i don't follow any strict rules.

wader commented 2 years ago

Wops accidentlly submitted above comment before i was done writing... editing now :) ok done editing

binoculars commented 2 years ago

The cache limits for gha are currently 10 GB and 7 days which seems pretty reasonable.

Not sure about the .so files—I haven’t tested these features yet, although I had frei0r in an old build.

Yes libcaca failed on the build but not configure, so I tried pushing that while testing it failed so I had to remove it.

wader commented 2 years ago

Adding caching https://github.com/wader/static-ffmpeg/pull/253. Not as useful without more stages but much better than nothing

wader commented 2 years ago

BTW know if there are any security concern with using a shared cache between PR and other builds? if content addressable i would guess not?

binoculars commented 2 years ago

Not that I'm aware of. Works like a charm if you only build master branch, but I'm just too paranoid to let it build without a manual review first.

wader commented 2 years ago

Ok, will probably sync some other things from your fork when i get time, thanks again for working on it! let's keep in touch!