Closed ToshY closed 2 weeks ago
ToshY
commented
2 weeks ago I'll close this because I think the workaround suffices for now.
RUN set -ex \
&& apt-get update \
&& apt install -y fontconfig
# Custom config containing the fonts directory "/usr/localx/share/fonts"
COPY <<EOF /etc/fonts/conf.d/50-custom.conf
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "urn:fontconfig:fonts.dtd">
<fontconfig>
<dir>/usr/localx/share/fonts</dir>
</fontconfig>
EOF
wader
commented
2 weeks ago Hey, thanks for reporting and figure out a workaround! Maybe something about this could be added to the "Fonts usage with SVG or draw text filters etc" section in the README? btw how does this work, will ffmpeg via libfontconfig (?) somehow end up executing fontconfig to update caches etc? or when does that happen? and guess another concern would be how stable the cache format is between fontconfig versions? (version of fontconfig linked in the ffmpeg binary vs version of fontconfig binary is installed)
ToshY
commented
2 weeks ago Hey @wader
Maybe something about this could be added to the "Fonts usage with SVG or draw text filters etc" section in the README?
Agreed, adding something like the beforementioned workaround to the README is a good idea. While I normally would not mind to create a PR for this myself, the problem I'm having has to do with your follow-up questions: how does this work?
And the answer to that is: I don't really know. 🤷♂️
I've been spending the evening with a bit of debugging/trial-and-error in order to try to understand how/what/where, and I'll just put down what I tried here.
As you've might have guessed from my previous example, I want to use ffmpeg/ffprobe binaries in a multistage build. While I initially thought copying both binaries would be enough (like stated the README), I totally overlooked that fonts are major part of rendering subtitles. As source files sometimes are missing embedded fonts, it's nice that FFmpeg can fallback to fonts installed on system. As a sidenote, the actual rendering of subtitles is done in this case by libass.
Basically the original Dockerfile I'm currently working on boils down to this:
FROM python:3.11-slim-bookworm AS base
# Do stuff here.
FROM base as ffmpeg
COPY --from=mwader/static-ffmpeg:7.0.1 /ffmpeg /usr/bin/
COPY --from=mwader/static-ffmpeg:7.0.1 /ffprobe /usr/bin/
FROM ffmpeg AS prod
# More stuff here.Now with the workaround mentioned earlier it looks like this:
FROM python:3.11-slim-bookworm AS base
RUN <<EOT bash
set -ex
apt-get update
apt install -y fontconfig
apt-get clean
rm -rf /var/lib/apt/lists/*
EOT
COPY <<EOF /etc/fonts/conf.d/50-custom.conf
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "urn:fontconfig:fonts.dtd">
<fontconfig>
<dir>/usr/localx/share/fonts</dir>
</fontconfig>
EOF
# Do stuff here.
FROM base as ffmpeg
COPY --from=mwader/static-ffmpeg:7.0.1 /ffmpeg /usr/bin/
COPY --from=mwader/static-ffmpeg:7.0.1 /ffprobe /usr/bin/
FROM ffmpeg AS prod
# More stuff here.Like stated earlier, I'm fine with this as it seems to work like intended.
At this point I started debugging the Dockerfile (of mwader/static-ffmpeg:7.0.1) further, which lead to me having several "problems" with this image when trying to figure stuff out, as I'm not used to images not having a shell. Building the image locally and replacing scratch with alpine for the final1 stage to atleast have a shell and then I installed fontconfig-dev fontconfig-static (same install that were also done earlier in build stage).
FROM alpine:3.20.0 AS final1
COPY --from=builder /usr/local/bin/ffmpeg /
COPY --from=builder /usr/local/bin/ffprobe /
COPY --from=builder /versions.json /
COPY --from=builder /usr/local/share/doc/ffmpeg/* /doc/
COPY --from=builder /etc/ssl/cert.pem /etc/ssl/cert.pem
COPY --from=builder /etc/fonts/ /etc/fonts/
COPY --from=builder /usr/share/fonts/ /usr/share/fonts/
COPY --from=builder /usr/share/consolefonts/ /usr/share/consolefonts/
COPY --from=builder /var/cache/fontconfig/ /var/cache/fontconfig/
RUN apk add --no-cache fontconfig-dev fontconfig-staticNow I was able to get in the shell and verify that fc-list and fc-cache work. A minor setback is that is throws errors regarding no writable cache directories:
/usr/share/fonts $ fc-cache -f
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
/usr/share/fonts/dejavu: failed to write cache
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
/usr/share/fonts/font-awesome: failed to write cache
Fontconfig error: No writable cache directories
/usr/share/fonts/inconsolata: failed to write cache
Fontconfig error: No writable cache directories
/usr/share/fonts/misc: failed to write cacheEven though it shows these errors, it does not seem to impact the font rendering (still finds them correctly). Not sure if this can be resolved as well (cannot find good sources on how to fix this).
This one I'll add just for those want to use mwader/static-ffmpeg:7.0.1 standalone (and not in multistage build), to show that you can mount your custom fonts directory and fonts config if you just mount the volumes.
docker run -i --rm -u $(id -u):$(id -g) -v "$PWD/input:$PWD" -v "$PWD/fonts:/usr/localx/share/fonts" -v "$PWD/50-custom.conf:/etc/fonts/conf.d/50-custom.conf" -w "$PWD" mwader/static-ffmpeg:7.0.1 -i ...Like I said, this is nice for those who just want to run it as-is but with custom fonts.
TLDR;
I think the workaround is a somewhat "clean solution" because it will not require any changes in your image, and it can be added without to much effort in my own previous build stage.
Still, I haven't figured out the fontconfig writable cache directories issue, so if you have any insights on why that might happen (or how to resolve it) that would great.
wader
commented
2 weeks ago As you've might have guessed from my previous example, I want to use ffmpeg/ffprobe binaries in a multistage build. While I initially thought copying both binaries would be enough (like stated the README)
Ah good point, maybe the README should also have a note to look below about additional files like font and ssl certs.
At this point I started debugging the Dockerfile (of mwader/static-ffmpeg:7.0.1) further, which lead to me having several "problems" with this image when trying to figure stuff out, as I'm not used to images not having a shell
Sorry about that :) i've usually just trimmed away the last part the dockerfile to keep the build env when i want to test something, other solutions is copy out the binaries to the host or as you did.
Now I was able to get in the shell and verify that fc-list and fc-cache work. A minor setback is that is throws errors regarding no writable cache directories:
Weird, your running as root in the container or as some user? but i wonder if things work because it just skips using a cache and it will just be a bit slower?
This one I'll add just for those want to use mwader/static-ffmpeg:7.0.1 standalone (and not in multistage build), to show that you can mount your custom fonts directory and fonts config if you just mount the volumes.
Nice one! did this also produce warnings?
Thanks for all the research! I'll will try to distill it down something to be included in the README. Your more than welcome to do a draft PR also.
ToshY
commented
2 weeks ago Sorry about that :) i've usually just trimmed away the last part the dockerfile to keep the build env when i want to test something, other solutions is copy out the binaries to the host or as you did.
No reason for apology, I learned something new again 🙂
Weird, your running as root in the container or as some user?
Great observation! That's it. I run them as non-root currently by passing -u $(id -u):$(id -g). If I provide -u root, and then run fc-list or fc-cache -v, it no longer shows the warnings. 👍
...but i wonder if things work because it just skips using a cache and it will just be a bit slower?
That would be my guess as well, even though I haven't really find a noticable performance difference yet between using cache or not.
Nice one! did this also produce warnings?
Yes, but also if run as non-root.
I wonder if I add a <cachedir> entry that is writable for the user maybe it will resolve the warnings as well.
wader
commented
2 weeks ago Sorry about that :) i've usually just trimmed away the last part the dockerfile to keep the build env when i want to test something, other solutions is copy out the binaries to the host or as you did.
No reason for apology, I learned something new again 🙂
🥳
Weird, your running as root in the container or as some user?
Great observation! That's it. I run them as non-root currently by passing
-u $(id -u):$(id -g). If I provide-u root, and then runfc-listorfc-cache -v, it no longer shows the warnings. 👍
Aha good 👍 I did a quick digg and it seems to be FcDirCacheRead that will write a cache directory and going backwards from where it's used https://gitlab.freedesktop.org/fontconfig/fontconfig/-/blob/main/src/fccfg.c#L509 i get a feeling any user of libfontconfig might end up writing to the cache? but not sure.
I wonder if I add a
<cachedir>entry that is writable for the user maybe it will resolve the warnings as well.
Mm i would guess that should work.
ToshY
commented
2 weeks ago i get a feeling any user of libfontconfig might end up writing to the cache? but not sure.
I think you're correct with that. As non-root user I found it is able to write cache to the default /var/cache/fontconfig as long as the directory is chmod with 777.
RUN <<EOT bash
set -ex
chmod -R 777 /var/cache/fontconfig
EOTTrying to chmod to e.g. 776 (or lower) will show similar permission denied warnings when running fc-cache -v.
/var/cache/fontconfig: invalid cache file: d589a48862398ed80a3d6066f4f56f4c-le64.cache-8
/var/cache/fontconfig/d589a48862398ed80a3d6066f4f56f4c-le64.cache-8: Permission deniedI can start with drafting up a PR that shows how to extend the Dockerfile in order to use custom fonts directory.
Problem
With FFmpeg
7.0.1image, the currently installed fontconfig for is2.15.0(as seen from theversion.json), which has a/etc/fonts/fonts.conflike so:
```xml/etc/fonts/fonts.confIf I want to use custom font config to allow for more font directories (e.g.
/usr/localx/share/fonts), I have to create a custom config in/etc/fonts/conf.d. The problem however is, that even if I add a custom config, it will not be used as it seems the image is missing fontconfig binaries (don't seem to be copied in final1 stage).So even if I mounted a custom config now, when running FFmpeg this will lead to fonts not being found:
A workaround is to manually install
fontconfigand no longer copy the/etc/fontsdirectory.When running FFmpeg now I see the font being found:
Solution
Add the fontconfig binaries to the
final1stage so they can be copied as well (?)