Unable to install custom integration packages after upgrade to 2024.10

pkostenko commented 1 month ago

The problem

After upgrading to 2024.10 with new uv package manager, home assistant can no longer install custom integration dependencies when running in root-less docker container.

When running the container, a custom UID is passed: --user "200:200".

From the logs, it looks like uv is trying to use directories (/.cache/, /tmp/) that can be created only under root.

Would it be possible to move the uv cache under the /config/ path instead, which would allow it to inherit the regular permissions?

What version of Home Assistant Core has the issue?

core-2024.10.2

What was the last working version of Home Assistant Core?

core-2024.9.3

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Google Calendar

Link to integration documentation on our website

https://www.home-assistant.io/integrations/google

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.util.package
Source: util/package.py:150
First occurred: 04:23:07 (3 occurrences)
Last logged: 04:23:07

Unable to install package ical==8.2.0: error: failed to create directory `/.cache/uv` Caused by: Permission denied (os error 13)

Unable to install package ical==8.2.0: error: failed to create file `/tmp/uv-1b696e695b7c17a7.lock` Caused by: Permission denied (os error 13)

Additional information

No response

home-assistant[bot] commented 1 month ago

Hey there @allenporter, mind taking a look at this issue as it has been labeled with an integration (google) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `google` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign google` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

google documentation google source _{^{(message by IssueLinks)}}

allenporter commented 1 month ago

My impression is that changing around the user that runs in the container isn't something officially supported, so I don't think I can really help if this is changing the official container in this way. (I don't have an opinion on this myself, just sharing my understanding from previous discussions and i'm not in a good position to debate this really. I can't find the previous discussions on this but #7872 is maybe close.)

Someone discusses a workaround in #127812 however, that you may find useful. Respecfully, closing and de-duplicating against that bug..

rmi1974 commented 1 month ago

UV_CACHE_DIR from https://github.com/home-assistant/core/issues/127812 alone won't work. There is still the annoying part that a root-owned lockfile in /tmp exists which 'uv' when being run as "homeassistant" user can't remove, hence it fails.

https://github.com/astral-sh/uv/blob/2901a8d6cc8f4f8c43829a3c9fc7c1d4879a4d76/crates/uv-python/src/environment.rs#L277

Even if one removes the /tmp/xxx.lock file from the container, the custom components installation fails because it tries to remove/refresh packages from global package cache which is root owned:

Within container:

supernova:/config# ls -lsa /tmp
total 0
     0 drwxrwxrwt    1 root     root             6 Oct 13 10:58 .
     0 drwxr-xr-x    1 root     root            62 Oct 13 10:58 ..
     0 -rw-r--r--    1 root     root             0 Oct 11 18:33 uv-1b696e695b7c17a7.lock

supernova:/config# rm /tmp/*.lock

supernova:/config# exit

Restart container:

homeassistant  | 2024-10-13T09:14:27.473191000Z 2024-10-13 11:14:27.472 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package icalevents!=0.1.28: error: failed to remove file `/usr/local/lib/python3.12/site-packages/httplib2-0.20.4.dist-info/INSTALLER`
homeassistant  | 2024-10-13T09:14:27.473321000Z   Caused by: Permission denied (os error 13)

" error: failed to remove file /usr/local/lib/python3.12/site-packages/httplib2-0.20.4.dist-info/INSTALLER"

Or manually simulating the problem within container:

supernova:/config# su homeassistant
/config $ cd

~ $ uv pip install icalevents!=0.1.28 --index-strategy unsafe-first-match --upgrade --constraint /usr/src/homeassistant/homeassistant/package_constraints.txt
Resolved 7 packages in 720ms
Prepared 3 packages in 87ms
error: failed to remove file `/usr/local/lib/python3.12/site-packages/httplib2-0.20.4.dist-info/INSTALLER`
  Caused by: Permission denied (os error 13)

~ $ exit
supernova:/config# uv pip install icalevents!=0.1.28 --index-strategy unsafe-first-match --upgrade --constraint /usr/src/homeassistant/homeassistant/package_constraints.txt
error: failed to create file `/tmp/uv-1b696e695b7c17a7.lock`
  Caused by: Permission denied (os error 13)

supernova:/config# ls -lsa /tmp
total 0
     0 drwxrwxrwt    1 root     root            38 Oct 13 11:39 .
     0 drwxr-xr-x    1 root     root            74 Oct 13 11:07 ..
     0 -rw-r--r--    1 homeassi homeassi         0 Oct 13 11:39 uv-1b696e695b7c17a7.lock

supernova:/config# ls -lsa /usr/local/lib/python3.12/site-packages/httplib2-0.20.4.dist-info/INSTALLER
     4 -rw-r--r--    1 root     root             2 Oct 11 18:32 /usr/local/lib/python3.12/site-packages/httplib2-0.20.4.dist-info/INSTALLER

If one re-creates the container, it gets even worse (/tmp/ root-owned lockfile problem again).

One workaround is to run 'uv' as root within the container and manually do package installs for each component

supernova:/config# rm /tmp/*

supernova:/config# uv pip install icalevents!=0.1.28 --index-strategy unsafe-first-match --upgrade --constraint /usr/src/homeassistant/homeassistant/package_constraints.txt
Resolved 7 packages in 929ms
Prepared 3 packages in 44ms
Uninstalled 2 packages in 19ms
░░░░░░░░░░░░░░░░░░░░ [0/3] Installing wheels...                                                                                                                        warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 3 packages in 12ms
 - httplib2==0.20.4
 + httplib2==0.22.0
 - icalendar==6.0.0
 + icalendar==5.0.13
 + icalevents==0.1.29

supernova:/config# uv pip install boschshcpy==0.2.95 --index-strategy unsafe-first-match --upgrade --constraint /usr/src/homeassistant/homeassistant/package_constraints.txt
Resolved 12 packages in 928ms
Prepared 2 packages in 70ms
Uninstalled 2 packages in 3ms
░░░░░░░░░░░░░░░░░░░░ [0/2] Installing wheels...                                                                                                                        warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 2 packages in 3ms
 - boschshcpy==0.2.91
 + boschshcpy==0.2.95
 - getmac==0.9.4
 + getmac==0.9.5

(repeat for each 'homeassistant.util.package' error)

This obviously completely defeats the container user concept is will immediately break next time a HACS component is upgraded or the container is recreated.

Not sure which PR introduced the problem?

https://github.com/home-assistant/core/pull/125808

or something earlier?

The breaking change is mentioned here:

https://www.home-assistant.io/blog/2024/10/02/release-202410/ https://developers.home-assistant.io/blog/2024/04/03/build-images-with-uv/

On each upgrade, all dependencies needed for custom integrations are now downloaded and installed. We are now using uv internally to download and install dependencies, which is a faster and more efficient way to download and install dependencies. If you’re not aware, uv is what makes our release process so fast nowadays, and [@edenhaus](https://github.com/edenhaus) has done an interesting [developer backstory](https://developers.home-assistant.io/blog/2024/04/03/build-images-with-uv/) on our move from pip to uv (saving us 200+ hours of execution time a month). [@edenhaus](https://github.com/edenhaus) has also implemented uv at runtime as well.

This speeds up the upgrade process, especially for custom integrations with many dependencies, or installations with many custom integrations. This change is fully transparent to the user, and no action is required. Just sit down, relax, and enjoy the faster upgrade process!

Apparently it was not properly tested in Docker integration scenarios which run non-root docker users.

All in all this is a very unfortunate development change that HA installations using Docker with non-root container users are now broken. In fact those setups are fairly common and recommended industry standard/practice:

https://www.redhat.com/en/blog/understanding-root-inside-and-outside-container https://dev.to/kfir-g/securing-docker-non-root-user-best-practices-odb

Given the size/scope/goal of this project I find this rather unsatisfactory.

allenporter commented 1 month ago

Yes, my primary point was that my impression is the home assistant container doesn't claim to support changing the user. (You won't like that answer.)

seidler2547 commented 1 month ago

I was able to make it work as before by changing my Dockerfile to also chown /tmp to the user:

FROM ghcr.io/home-assistant/home-assistant:latest

RUN apk add iputils
RUN addgroup -g 1000 homeassistant ; adduser -s /bin/bash -D -h /home/homeassistant -G homeassistant homeassistant ; chown 1000:1000 /home/homeassistant ; chown 1000:1000 /usr/local/lib/python3*/site-packages ; chown 1000:1000 /usr/local/bin /tmp /tmp/*

goodnewz commented 1 month ago

@rmi1974 An alternative to @seidler2547 's solution, and one that you might find more appealing, is to specify a site_package directory for Python that is not the default directory your user does not have permission to use. Python supports this through the PYTHONNOUSERSITE environmental variable. I set mine in the config folder /config/python_userdir.

While this worked until 2024.10.0, it did not after. The funny part is that the fix provided in #125808/2077 works if applied manually by logging into the container from docker, but not when it runs by home assistant at startup. For me, it tries to use the system site package directory, not the one I defined with the environmental variable. Perhaps something in the logic of #125808 for me.

Remember to add --python --target options when running the install command. In my case, this looks something like this:

uv pip install hubspace-async==0.4.1 --index-strategy unsafe-first-match --upgrade --constraint /usr/src/homeassistant/homeassistant/package_constraints.txt --python /usr/local/bin/python --target /config/python_userdir/lib/python3.12/site-packages

aptalca commented 1 month ago

One alternative is to use a venv in the bind mounted folder, but that's also not working properly because uv does not consider system packages when you do a uv pip list or uv pip install so whatever dependency a custom integration needs, uv will install the whole dependency tree in that venv, even if those packages are already installed in the system path. Not only there will be duplication, but the uv installed packages may override the HA versions.

https://github.com/astral-sh/uv/issues/4466

Also, the main issue is that uv does not support the --user flag and behavior that pip supports where it will install packages in the user's home folder if run by a non-root user, but does happily consider system packages when doing so. https://github.com/astral-sh/uv/issues/2077

Not a fan of uv in this context so far. Sure it installs the 1400 packages that HA needs super fast during build time, but it causes tons of issues runtime.

rmi1974 commented 1 month ago

There is another way. I totally forgot that my docker container injects/bind mounts the runner script from:

https://github.com/tribut/homeassistant-docker-venv

    volumes:
    - ${DOCKER_HOST_VOLUME_ROOT}/homeassistant/config:/config
    - ${DOCKER_HOST_VOLUME_ROOT}/homeassistant/config/docker/run:/etc/services.d/home-assistant/run
...

The repo was not auto-updated on my docker host for a long time. Looks like they have a fix for this now:

https://github.com/tribut/homeassistant-docker-venv/issues/36 https://github.com/tribut/homeassistant-docker-venv/commit/996023803209d805fd30eb3467af94b0815a0acc

Home Assistant moved to `uv` in 2024.10. And they set `UV_SYSTEM_PYTHON=true` in their Docker image. This causes several errors (see https://github.com/tribut/homeassistant-docker-venv/issues/36) during start up when extra packages are installed (for example, by HACS).

We fix this by setting `UV_SYSTEM_PYTHON=false` after creating a venv.

This works well, it doesn't need UV_CACHE_DIR nor site package trickery:

supernova:~# ls -lsa /var/tmp/homeassistant-venv/lib/python3.12/site-packages/
total 320
     4 drwxr-xr-x   46 homeassi homeassi      4096 Oct 14 09:17 .
     0 drwxr-xr-x    3 homeassi homeassi        27 Oct 14 09:16 ..
   244 -rwxr-xr-x    2 homeassi homeassi    246081 Oct 14 09:17 _cffi_backend.cpython-312-x86_64-linux-musl.so
     4 drwxr-xr-x    3 homeassi homeassi      4096 Oct 14 09:17 boschshcpy
     0 drwxr-xr-x    2 homeassi homeassi       143 Oct 14 09:17 boschshcpy-0.2.95.dist-info
     0 drwxr-xr-x    3 homeassi homeassi        74 Oct 14 09:17 cachetools
     0 drwxr-xr-x    2 homeassi homeassi       119 Oct 14 09:17 cachetools-5.3.0.dist-info
...

Re-created my HA container and it installed/updated all HACS plugins and depds automagically. Not sure if overriding UV_SYSTEM_PYTHON to false is the long term way to go though.

rmi1974 commented 3 weeks ago

Upgrade to HA 2024.11 broke stuff again after another uv related change .

Caused by https://github.com/home-assistant/core/pull/128371

If you are using docker + virtual env (https://github.com/tribut/homeassistant-docker-venv) you need to update and then recreate the HA container.

https://github.com/tribut/homeassistant-docker-venv/issues/38

home-assistant / core