Container fails if UID:GID exist in underlying image

[codefaux]

Greetings. I haven't been able to use this yet, but I'm eager.

I tried to deploy it using the UID/GID which conflicts with the pre-existing users group.

This causes the "adduser" and/or "addgroup" command(s) to fail.

I could fork and branch and commit and create a pull request, but the init script only needs a few changes, so I just mounted over the entrypoint.sh script with my modified version.

The original code:

• blindly create a group with a GID which may be used with another one, then only if that succeeds, blindly create a user (and their home directory, which you've already created and later take permission of again) and assign it to the group we just created

  addgroup --gid $GID ydl_api_ng && useradd --uid $UID --gid ydl_api_ng ydl_api_ng

The new code:

• Check if group $GID exists. If it does, rename it so we can use it by name. If not, make it.
• Check if user $UID exists. If it does, rename it so we can use it by name. If not, make it.

  
  getent group $GID > /dev/null
  if [ $? -eq 0 ]; then
  groupmod $(id --name --group $GID) -n ydl_api_ng
  else
  addgroup --gid $GID ydl_api_ng
  fi

getent passwd $UID > /dev/null if [ $? -eq 0 ]; then usermod $(id --name --user $UID) -l ydl_api_ng else useradd --uid $UID --gid ydl_api_ng ydl_api_ng fi

New script:

!/bin/bash

echo ~~~ ydl_api_ng echo ~~~ Revision : $GIT_BRANCH - $GIT_REVISION echo ~~~ Docker image generated : $DATE

mkdir -p /app/logs /app/downloads /app/params /app/tmp /home/ydl_api_ng /app/data /root/yt-dlp-plugins /app/cookies/ cp -n /app/setup/* /app/params/ touch /app/data/database.json ln -s /app/data/database.json ./database.json

if [ "$FORCE_YTDLP_VERSION" == "" ]; then echo --- Upgrade yt-dlp to the latest version --- pip3 install yt-dlp --upgrade else echo --- Force yt-dlp version $FORCE_YTDLP_VERSION --- pip3 install yt-dlp==$FORCE_YTDLP_VERSION --force-reinstall fi

pip3 install -r /app/params/hooks_requirements

getent group $GID > /dev/null if [ $? -eq 0 ]; then groupmod $(id --name --group $GID) -n ydl_api_ng else addgroup --gid $GID ydl_api_ng fi

getent passwd $UID > /dev/null if [ $? -eq 0 ]; then usermod $(id --name --user $UID) -l ydl_api_ng else useradd --uid $UID --gid ydl_api_ng ydl_api_ng fi

chown $UID:$GID /app/logs /app/downloads /home/ydl_api_ng /app/tmp /app/data /app/data/database.json /app/cookies /root/yt-dlp-plugins chmod a+x /root/ entrypoint.sh

if [ "$DISABLE_REDIS" == "false" ]; then cat <>/app/supervisord_workers.conf [supervisord]

[program:worker] command=rq worker ydl_api_ng -u "redis://ydl_api_ng_redis:6379" process_name=%(program_name)s-%(process_num)s numprocs=$NB_WORKERS directory=. stopsignal=TERM autostart=true autorestart=true user=$UID EOT

supervisord -c /app/supervisord_workers.conf -l /app/logs/supervisord_workers.log -j /app/tmp/pid_api -u ydl_api_ng -e $LOG_LEVEL

cat <>/app/supervisord_programmation.conf [supervisord]

[program:programmation] command=python3 programmation_daemon.py process_name=%(program_name)s-%(process_num)s numprocs=1 directory=. stopsignal=TERM autostart=true autorestart=true user=$UID EOT

supervisord -c /app/supervisord_programmation.conf -l /app/logs/supervisord_programmation.log -j /app/tmp/pid_programmation -u ydl_api_ng -e $LOG_LEVEL fi

if [ "$DEBUG" == "DEBUG" ]; then echo ~ Launching DEBUG mode ~ su ydl_api_ng -c "uvicorn main:app --reload --port 80 --host 0.0.0.0" else su ydl_api_ng -c "python3 main.py" fi

[codefaux]

My solution above is imperfect and requires revision, somehow it winds up with gid 65535 sometimes.

Also, the Dockerfile does not expose variables, ports or volumes, which makes it difficult to deal with via inspection.

So, I'll be doing the usual fork, branch, commit, pr thing for a general Dockerfile cleanup (container init is not very graceful) if I can find the time.

I'll probably find the time, lol

[Totonyus]

Thank you for your proposition. I tested and pushed your code on the develop branch. You can use the preview docker image to use the develop branch. (This is the image I use myself).

I'll will obviously be very glad if you make an optimisation of the dockerfile :) (I did my best)

[Totonyus]

Note the build of the preview failes for an unknown reason (not related to the changes). I have to find how to correct that.

[codefaux]

No worries! I've been on a binge lately of obsessing over random ideas, finding people with the brains to get them running, and improving them how I can, and I "only" have around 60 containers currently running across my homelab, so I've had above average exposure to the various ways things can be done and tend to break lol.

It's always bugged me to have to remember the various videos released by the channels I watch, and I have Emby to manage my media, so I'm working on making Huginn pull episodes. My previous hack was forcing Huginn to install yt-dlp and a static binary of ffmpeg into its running container, then downloading, but it would time out or single-thread and stall while waiting for the download. Your API server here is a fantastic solution. I'm curious -- why did you build it?

[Totonyus]

The main purpose was to be able to launch livestream recording on my nas and not on my computer.

[Totonyus]

Found the problem with the image build, corrected in 3188d029ff8d63927ea6cf1d70957014f30e8257

[codefaux]

Adding Cargo and fixing the fastapi version is definitely not necessary, the container still builds as it was when I cloned it initially.

[Totonyus]

It seems there is a problem for arm containers

[codefaux]

We've got three problems.

• ffmpeg-static doesn't exist for arm64
• psutil is a shitty Python package which requires compiled binary C code to execute

I can manually download ffmpeg's static binaries and unpack them.

The requirement for psutil means we need to install a package. python:3.12-bookworm (and the other non-Alpine tags) use Debian as an underlying Linux distro. Debian bookworm and bullseye provide python3.9 -- which means if we use 'apt' to install python3-psutil, apt will replace the Python from the docker image with Debian's awfully-outdated python3.9 and bloat the image. This means there's literally negative benefit to starting with python:3.12-bookworm or, truly, ANY python:* image based on Debian and we should just start with a Debian (or other) baseimage.

Installing psutil on Alpine is cake-easy, and does not replace the entirity of Python with a worse version. However, this changes a lot of the entrypoint code, especially regarding arbitrary UID:GID execution.

Are you OK with changing from Debian to Alpine on the image?

[Totonyus]

Hi.

Thank you for your participation :)

However I can't manage to make things work with how you installed ffmpeg :

[25-05-24 15:51:25][youtube-dlp][ERROR] ERROR: You have requested merging of multiple formats but ffmpeg is not installed. Aborting due to --abort-on-error

I tried a lot of things but it just doesn't work when ffmpeg is needed. Did you tried this case ?

Also : I tried switching to alpine but it just causes too much problems that I'm able to handle for now. I just made the installation of gcc only for arm build.

For now (with ffmpeg installed with apt) the image for amd64 is 767MB

[codefaux]

I tried a lot of things but it just doesn't work when ffmpeg is needed. Did you tried this case ?

My yt-dlp configuration specifies the location of ffmpeg as I copied it from another system I had cobbled together. I tested that ffmpeg worked from the console and assumed the rest would fall into place. I did not think to check if it worked via default configuration, as I assumed ffmpeg-static installed it into a PATH-available location and "it worked for me" lol. It's likely just put ffmpeg into a strange place, but it would be easier for you to maintain a "same build for all platforms" solution so I'll try to aim for that, knowing it's a design goal.

I was already working to resolve this, as pip ffmpeg-static does not work on arm. The downloaded ffmpeg static binaries via my current route add ~100mb to the base image (50mb each for ffmpeg and ffprobe static binaries) and it takes only as long as downloading and unpacking the file, and adds zero* other dependencies. I think my final image was below 350mb on all platforms, and the build itself was dramatically shorter not having to install a compiler suite, compile things, and install ffmpeg and the deps for it.

Note: I'm not advocating or arguing that one way is better than the other, I'm providing this simply for information because I'm one of those infodump people:

Re: Alpine vs Debian; The package manager in Alpine is apk add not apt get and there's no need to update at all, or clean up a cache as apk doesn't keep a local package database to update or remove. It doesn't come with bash so your shell is sh which means some of the tests and syntax change.

Some of the issues come from your entrypoint.sh script using a mix of commands from different packages to manage the user/group data.

adduser and deluser are from the Debian-derived adduser package. The adduser package only exists on Debian-derived Linux distros and is, according to documentation intended to be used by the local administrator in lieu of the tools from the 'useradd' suite, and they provide support for easy use from Debian package maintainer scripts, functioning as kind of a policy layer to make those scripts easier and more stable to write and maintain.

The useradd userdel groupadd groupdel groupmod and usermod and so forth exist from shadow-utils which is typically considered core Linux and available on virtually every Linux platform. The things which are breaking are things which are being done "the Debian way" rather than "the Linux" way.

Generally speaking, the difference only matters if you want to write for generalized Linux platforms instead of Debian-derived Linux. Different packages (Alpine's busybox, for example) implement useradd and other tools in their most basic, least feature-supported ways, as they are designed to be slim, not approachable.

ANYWAY, I'm mostly just rambling to write down the information somewhere permanent, lol...with the info dump over..

I'll stick to Debian for the container base, since it's what you're familiar with. I'll submit another PR once I've had time to kick something together. Thanks again for your work bringing the thing to life in the first place.

[Totonyus]

Hi !

I tried a lot of things to make ffmpeg works as you installed it but nothing worked (symlink, copy, specify location to yt-dlp...). If you can give me the configuration that worked for you I can see how to implement that.

[codefaux]

OK - I'll break down my suggestions, as you seem more driven to work on it right now tham me lol

The way your container works, I suggest building from debian:bookworm or debian:bullseye -- using debian:*-slim won't save much since you're about to install a bunch of packages and :bullseye is already only ~124mb on x64.

I suggest this because when installing things with apt, it's going to update what was preinstalled on the container, which means an extra layer (it doesn't delete from layers, it can only say "ignore what you downloaded, use this instead"), and that also means most of what you get in the python: containers gets replaced. They use the same base images as I'm suggesting, so it's smaller/faster to use a less complete image base, in your case.

Where possible, use pip to install in dockerfile if it's a consistent requirement, or at runtime if you need the most up to date versions or it's dynamic. If a package requires compiling a wheel (aka needs gcc) see if apt can install it instead, and do it in the dockerfile when possible. If you're ever going to run apt during container runtime, you may as well not run it during the dockerfile, and you may as well not clean it up since it'll be re-run at the next runtime and just take longer. Usability vs space trade-offs here, but on most platforms usability wins here.

The python3-psutil apt package covers the only compiled wheel requirement I'm aware of with your container, but I figured your requirements might change in the future so I should mention how I got there. Debian's apt package archives are binary on all platforms, and the resulting image size saving + speedup will be immense. Speaking of apt cleanup stage;

The Docker best practices list the following;

RUN apt-get update && apt-get install -y \
    package-bar \
    package-baz \
    package-foo  \
    && rm -rf /var/lib/apt/lists/*

https://docs.docker.com/develop/develop-images/instructions/

I believe the image size savings here will leave your container under 300mb on all platforms, but I haven't finished my local write-up.

Are you manually running commands in a base image to test things before building with Docker? If you don't know how, I can teach you and it'll save you SO. MUCH. EFFORT. (it boils down to docker run -ti --rm debian:bookworm bash to remove it when done, and docker run -ti --name testcontainer debian:bookworm bash to keep the container for future use..and obviously whichever volumes and ports you need but I feel like it's likely you know so I won't spend much time here.)

The part you're probably here for: My current suggestion to install ffmpeg via static build requires two things. First, a downloader and second is the apt xz-utils package. We rely on https://johnvansickle.com which has been pretty solid in my experience. He's been doing it since 2017, I'd suggest supporting their patreon if you see this container getting a lot of attention.

Here's how you would do it while attached to a running container;

apt update && apt install -y wget xz-utils && \
ARCH=$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) && \
RELEASE=$(wget https://johnvansickle.com/ffmpeg/release-readme.txt -q -O - | head -n 20 | sed -nE "s/.*build:.*ffmpeg-(.*)-${ARCH}-static.tar.xz/\1/p") && \
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-${ARCH}-static.tar.xz -O /ffmpeg.tar.xz && \
tar -xf /ffmpeg.tar.xz -C /tmp && \
install --mode=777 /tmp/ffmpeg-${RELEASE}-${ARCH}-static/ff{mpeg,probe} /usr/bin && \
rm /ffmpeg.tar.xz /tmp/ffmpeg-${RELEASE}-${ARCH}-static -rf

And to explain what I've done;

The ARCH line uses the arch command to get ..the arch.. but unfortunately it doesn't return it in the format we need so we 'sed' it twice with a simple substitution regex. I'm pretty sure i686, armhf and armel use the right string but it's easy to regex if you know the source and target strings.

RELEASE uses wget (smallest/least-dependant downloader) to pull the readme to stdout -q -O -, of which we read the first 20 lines by piping through head | head -n 20 |, and sed that for the version string we need using a capture regex. wget then pulls the binary replacing the arch string to a known-name/location, tar unpacks it to /tmp and we copy/chmod both ffmpeg and ffprobe in one command (replacing for release and architecture and using bash string magic*) to /usr/bin, then cleanup.

Feel free to ask any questions, I'm obviously more than eager to share my knowledge lol. Sorry for being so verbose, ideally this is helpful.

*bash string magic: touch test-{a,b} or touch test{a,b{1..5}} or touch a_{1..4}{,b{alpha,beta}} for all sorts of fun concatenation toys if you want to experiment. You can also do string manipulation like removing, replacing, etc. See https://tldp.org/LDP/abs/html/string-manipulation.html if you're into it.

[Totonyus]

Hi !

Thank you for your last comment.

As waited your way of installing ffmpeg helped me to find an anwser. In fact the static-ffmpeg modules doesn't really install ffmpeg. But when you launch the command static_ffmpeg you can notice those lines :

Download of https://github.com/zackees/ffmpeg_bins/raw/main/v5.0/linux.zip -> /usr/local/lib/python3.12/sitepackages/static_ffmpeg/bin/linux.zip completed.
Extracting /usr/local/lib/python3.12/site-packages/static_ffmpeg/bin/linux.zip -> /usr/local/lib/python3.12/sitepackages/static_ffmpeg/bin

The first launch just re-download the real ffmpeg then you can use static_ffmpeg command just like regular ffmpeg. So I created a symlink to ffmpeg and... It works ! In logs of ffmpeg you can say it comes from the same website (johnvansickle.com)

Then I had a revelation : I didn't really understand the concepts of docker containers and docker image. For now, in my personnal use, stopping a container was with the docker-compose down command (container destruction).

There is no real difference if a user downloads an image with ffmpeg installed or downloads an image that will download ffmpeg on launch. (Maybe difference of reliability ?)

So I just moved every pip install in the entrypoint script with a control to know if this is the first launch.

With all those optimisations :

• For amd build : ydl_api_ng latest 1e1688656400 About an hour ago 128MB
• For arm build : ydl_api_ng latest 7179a7e45a95 6 seconds ago 353MB

The first launch takes more time but it feels cleaner this way !

I tried using the debian base image you suggested and managed to make it runs but I had to install python in the dockerfile so the image is about 700MB...

I may try a few more things to unify amd and arm build but I'll probably not loose my sanity on this for the moment.

I'll put this on my server tomorrow to test it during a few days but all my local tests are great !

[Totonyus]

I just made something usable with the  python:3.12-alpine base image but the final image is 282MB (I still have to manually install gcc)

[codefaux]

Which python wheel is requiring compilation?

In my short testing I only found psutil, which is apk add py3-psutil on top of the python:3.12-alpine image.

Side note, apk add py3-psutil updates python to 3.12.3-r1 as an extra layer on top of the python:3.12 image, which still means using a python: image and then updating things is actually using more space and time than using a base distro instead. Alpine is just much better about it.

[Totonyus]

Maybe I missed something but the installation of py3-psutil isn't enough. I still have to put it as pip requirement.

[codefaux]

First, I apologize -- I haven't been testing all of this end-to-end because I've been short on time, so I've said some inaccurate things. I've been busy, and I'm not taking the time I should to test.

There is no real difference if a user downloads an image with ffmpeg installed or downloads an image that will download ffmpeg on launch. (Maybe difference of reliability ?)

If ffmpeg is installed in the image, the user doesn't have to wait for it to download and install every time they restart the container -- and if they're restarting the container, it's already been installed from the last time it was started. Not everyone destroys and recreates their containers every time -- most of us reboot servers once in a while, or use the "start" and "stop" commands. docker compose is not the only/primary way containers are used. If it made sense to build the container from scratch every time it was run, it would not be a docker image, it would be a dockerfile. "Everything to use the container" is generally expected to be in the container when it starts, the first time.

Also, in your entrypoint.sh script, every time the container starts up, it tries to add a new user and group. The last time it started, it added the same user and group -- so any time after the first time, part of the startup script fails, because the container was not destroyed, it was stopped and restarted.

Regarding installing ffmpeg via pip install static-ffmpeg -- I used static-ffmpeg myself in the past, but I didn't realize it was abandoned ages ago and broken, or I would not have recommended it again now. Upon looking deeper into it a few days ago (when you said it was not working, thus my new means of installing ffmpeg) I discovered that static-ffmpeg does not work on arm64, at least on my hardware. It installs, but the binary is in the wrong executable format. Further, on either platform, the binary installed after running static_ffmpeg is v5.0 and hasn't been updated for over two years. It is NOT the same as the webpage I used in the new method, not even close. It comes from zackees on github, not https://johnvansickle.com/ -- The binary downloaded by the script method I supplied most recently is v7.0, and has been updated very regularly.

Regarding my new method of installing a static build of ffmpeg, I noticed that I made a mistake in the version I pasted here.

When I put it here, I noticed a hard-coded amd64 string and converted it to a variable, but it's reading a text file which only shows the amd64 filename so it should have been hard-coded so when I pasted it here and "fixed" it I actually broke it. This method works, and has been tested on two arm64 platforms, as well as my amd64 system. I also removed the bash-specific part so it should run unmodified in both Debian and Alpine, which uses busybox's "sh" implementation as its shell.

ARCH=$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) && \
RELEASE=$(wget https://johnvansickle.com/ffmpeg/release-readme.txt -q -O - | head -n 20 | sed -nE "s/.*build:.*ffmpeg-(.*)-amd64-static.tar.xz/\1/p") && \
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-${ARCH}-static.tar.xz -O /ffmpeg.tar.xz && \
tar -xf /ffmpeg.tar.xz -C /tmp && \
install --mode=777 /tmp/ffmpeg-${RELEASE}-${ARCH}-static/ffmpeg /usr/bin && \
install --mode=777 /tmp/ffmpeg-${RELEASE}-${ARCH}-static/ffprobe /usr/bin && \
rm /ffmpeg.tar.xz /tmp/ffmpeg-${RELEASE}-${ARCH}-static -rf

^^^ That works on i686, amd64, arm64, armel, and armhf but I have not personally tested armel or armhf as I have no hardware on hand for it. It works in Alpine and Debian.

Regarding alpine vs debian:

• Alpine has up-to-date packages for py3-psutil and everything should work without gcc on Alpine if you rewrite your entrypoint.sh to function in Alpine instead of Debian.

• Debian hasn't updated their python3-psutil package in -FOREVER- and that's why you also have to reinstall it with pip, which is why it must be compiled when you use Debian. psutil is in fact the only package in your project which needs a wheel compiled on Debian, but since Debian hasn't bothered to update the package, the apt package for it is useless. It works without gcc on amd64, so I suggest only install gcc on arm.

Your container will not accept a change of UID/GID at startup which is how containers are expected to react. UID/GID should not need to be changed prior to building a container, most Docker users do not have the capacity or access to rebuild a container. It should change when passed UID/GID environment variables at container creation. I'm not sure how to fix it, and I've run out of motivation to try to make this work for now.