Custom integrations with requirements fail to start with 2022.7.X

[muhlba91]

The problem

I am using two integrations (hella_onyx, deebot_client) that have requirements according to their manifest.json. Since the upgrade of home assistant to 2022.7.X, both integrations fail to load:

2022-07-08 08:07:22 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration hella_onyx: Unable to import component: No module named 'onyx_client'
2022-07-08 08:07:25 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration deebot: Unable to import component: No module named 'deebot_client'

This is reopening #74597. I get that issues with custom components are not to be raised here. However, this is an infrastructure issue and not within multiple custom components using requirements in their manifest.json.

For example, installing the requirement with pip in the container (see https://github.com/py-smart-gardena/hass-gardena-smart-system/issues/125#issuecomment-1177218448) solves the problem. However, this is not a persistent change as I'd need to run this command for every requirement of a custom component whenever my container starts or the component needs updating. Installation of requirements should be handled by the infrastructure; in this case Home Assistant Core.

Please re-investigate/reopen #74597.

What version of Home Assistant Core has the issue?

core-2022.7.1

What was the last working version of Home Assistant Core?

core-2022.6.X

What type of installation are you running?

Home Assistant Core

Integration causing the issue

all custom components with requirements

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

I suspect a regression due to the Python upgrade in how to install/handle those requirements, or probably the directory pip installs them, and Python looks for. (see https://github.com/home-assistant/core/issues/74597#issuecomment-1178562739)

[frenck]

As said, please report issues with custom integrations with the custom integrations. As the Home Assistant project itself does not support custom integrations.

Many users use custom integrations, this has been extensively tested in beta as well; and is covered by our automated test.

For the sake of sanity I've just installed put the garden smart system integration linked above into my custom integrations folder:

And the requirements installed as expected.

Tested on a Home Assistant Blue (aarch64/arm64) Home Assistant 2022.8.0.dev20220708 Supervisor 2022.07.dev0703 Operating System 9.0.dev20220519 Frontend 20220708.0.dev - latest

[muhlba91]

I tried the same integration but once I tried configuring it I received:

2022-07-08 09:15:00 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration gardena_smart_system which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2022-07-08 09:15:18 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of py-smart-gardena==0.7.10
2022-07-08 09:15:24 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading configuration flow for integration gardena_smart_system: No module named 'gardena'

The same happens with any other custom component I try installing and configuring/using.

[frenck]

Can you provide details on your system? Like versions used, installation method, other custom integration in use, Docker images used (is applicable).

As in, I need something to reproduce the issue.

[muhlba91]

Installation method is in a Kubernetes pod/container using the ghcr.io/home-assistant/home-assistant:2022.7.1 image. There's no custom configuration or similar around that - I take the container and run it without a mount to my configuration directory /config.

Custom integrations in use are hella_onyx, version 3.0.0, and deebot, version 1.4.1. Both fail with the same error, and worked with 2022.6.X. I added the gardena integration (version 0.2.3) which fails with above posted error.

I see this line when starting home assistant which shows all enabled integrations, i believe:

2022-07-08 09:33:57 INFO (MainThread) [homeassistant.bootstrap] Setting up stage 2: {'mobile_app', 'automation', 'met', 'input_select', 'shell_command', 'prometheus', 'discovery', 'sun', 'sensor', 'input_text', 'tag', 'template', 'hella_onyx', 'history', 'logbook', 'energy', 'tts', 'webhook', 'influxdb', 'switch', 'input_number', 'fronius', 'blueprint', 'adguard', 'ring', 'timer', 'zone', 'scene', 'input_boolean', 'input_datetime', 'denonavr', 'my', 'ffmpeg', 'mqtt', 'heos', 'trace', 'nuki', 'counter', 'broadlink', 'device_tracker', 'forecast_solar', 'apple_tv', 'deebot', 'filesize', 'zwave_js', 'system_health', 'group', 'script', 'ipp'}

The deps directory in /config only contained Python 3.9 packages as 3.10 ones are now installed to /usr/local/lib/python3.10/site-packages now.

When enabling debug logging, I get this command to install the pip dependencies:

2022-07-08 08:53:59 DEBUG (SyncWorker_0) [homeassistant.util.package] Running pip command:
args=['/usr/local/bin/python3',
'-m', 'pip', 'install', '--quiet', 'deebot-client==1.5.1',
'--timeout', '60', '--upgrade',
'--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt',
'--find-links', 'https://wheels.home-assistant.io/musllinux/',
'--prefer-binary', '--user', '--prefix=']

For reference, a simplified version (remove annotations, labels, ...) of the Kubernetes Deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: home-assistant
  namespace: home-assistant
spec:
  replicas: 1
  template:
    spec:
      containers:
      - env:
        - name: TZ
          value: UTC
        image: ghcr.io/home-assistant/home-assistant:2022.7.1
        imagePullPolicy: IfNotPresent
        name: home-assistant
        ports:
        - containerPort: 8123
          name: http
          protocol: TCP
        volumeMounts:
        - mountPath: /config
          name: config
      serviceAccount: home-assistant-home-assistant
      serviceAccountName: home-assistant-home-assistant
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        persistentVolumeClaim:
          claimName: home-assistant-config

However, others in #74597 use podman or similar container orchestrators as well.

[bootc]

It looks like that pip command just isn't installing the pip modules in the deps directory as expected, instead they're going into /lib/python3.10/site-packages. For some reason they then can't be found by the intergrations when they are starting.

[bootc]

It's the --prefix= on the end that breaks it. Run that pip without that, but with PYTHONUSERBASE=/config/deps set, and the deps get installed correctly.

[muhlba91]

Thanks for the info - I‘ll have to check it on Monday as I‘m travelling at the moment.

However, if this is the solution, I‘m asking how this fix can be permanent as it’s not a solution to having to install all dependencies for each component (update) manually within the container first.

The above shown pip command is executed by HA Core itself (printed by using the debug log level) and I‘d expect it to execute correctly.

According to a comment in homeassitant.util.package the prefix= was added due to a work around necessary. Could the update to Python 3.10 render this unnecessary and cause the current behaviour?

[drod0258]

I'm having the same issue since updating to 2022.7 with the hubitat integration installed via HACS.

Commenting to track the status of this issue.

I run Home Assistant Core via the docker.io/homeassistant/home-assistant:stable docker image, in OKD 4.10 (Kubernetes 1.23, CRI-O 1.23).

[Preisschild]

Have the same issue with multiple custom integrations,

I use containerd as CRI with only /config persistent and use the homeassistant/home-assistant:2022.7.2 image.

[perosb]

Same here with containerd as runtime.

Using official docker image:

2022-07-09 14:55:43 ERROR (MainThread) [homeassistant.components.media_player] Error while setting up braviatv_psk platform for media_player
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 250, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/braviatv_psk/media_player.py", line 221, in async_setup_platform
    device = await hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/braviatv_psk/media_player.py", line 296, in __init__
    from braviapsk import sony_bravia_psk
ModuleNotFoundError: No module named 'braviapsk'
2022-07-09 14:55:46 ERROR (MainThread) [homeassistant.components.camera] The mjpeg platform for the camera integration does not support platform setup. Please remove it from your config.
2022-07-09 14:56:07 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration landroid_cloud: Unable to import component: No module named 'pyworxcloud'

Testing with linuxservers image:

2022-07-09 14:42:21 ERROR (SyncWorker_2) [homeassistant.util.package] Unable to install package pyworxcloud==2.1.10: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lib/python3.10'
Check the permissions.

WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

2022-07-09 14:43:28 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration landroid_cloud: Requirements for landroid_cloud not found: ['pyworxcloud==2.1.10'].

2022-07-09 14:44:03 ERROR (SyncWorker_1) [homeassistant.util.package] Unable to install package pySonyBraviaPSK==0.2.4: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lib/python3.10'
Check the permissions.

WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

2022-07-09 14:44:37 ERROR (SyncWorker_1) [homeassistant.util.package] Unable to install package pySonyBraviaPSK==0.2.4: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lib/python3.10'
Check the permissions.

WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

2022-07-09 14:45:13 ERROR (MainThread) [homeassistant.config] Platform error: media_player - Requirements for braviatv_psk not found: ['pySonyBraviaPSK==0.2.4'].

2022-07-09 15:03:32 INFO (SyncWorker_5) [homeassistant.util.package] Attempting install of pySonyBraviaPSK==0.2.4
2022-07-09 15:03:32 DEBUG (SyncWorker_5) [homeassistant.util.package] Running pip command: args=['/usr/local/bin/python3', '-m', 'pip', 'install', '--quiet', 'pySonyBraviaPSK==0.2.4', '--timeout', '60', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--find-links', 'https://wheels.home-assistant.io/musllinux/', '--prefer-binary', '--user', '--prefix=']

[onedr0p]

Thanks for the info @perosb this is the exact same thing happening to me. pip install from the container refuses to place libs into /config/deps/python3.10

[icaruseffect]

In #74405 described: Removing the "--prefix" line as described in the patch (link) by @synthead fixes the issue for me. Can anyone verify this?

[frenck]

Thanks, @muhlba91 for the reproduction that is helpful. 👍

It's the --prefix= on the end that breaks it. Run that pip without that, but with PYTHONUSERBASE=/config/deps set, and the deps get installed correctly.

@bootc Yes, that is related probably. The prefix needs to be either removed now, or, replaced by --install-option="--prefix=".

For this use case in this issue, removal should do. But it is in place folder an "older" issue with some distro. So removing it might affect users running Core on such distros.

I lean towards the latter one in that case, but not 100% sure if that works at this point (needs to be tested).

Edit to add a thought: Also, this behavior might be needed to be split more. As we might need a different behavior on Home Assistant Operating System & Supervised installs compared to the container version (They both use the same Docker image).

[onedr0p]

Yay for building a custom non-root home-assistant image? Seems like since /usr/lib is not writable in my case, it has no other choice but to use /config/deps defined from the PYTHONUSERBASE env var.

      containers:
      # Source: https://github.com/onedr0p/containers
      - image: ghcr.io/onedr0p/home-assistant:2022.7.2
        env:
        - name: PYTHONUSERBASE
          value: /config/deps

Defaulting to user installation because normal site-packages is not writeable
bash-5.1$ ls -la /config/deps/lib/python3.10/site-packages/
total 16
drwxrwsr-x    4 kah      kah           4096 Jul  9 11:44 .
drwxrwsr-x    3 kah      kah           4096 Jul  9 11:44 ..
drwxrwsr-x    3 kah      kah           4096 Jul  9 11:44 fullykiosk
drwxrwsr-x    2 kah      kah           4096 Jul  9 11:44 python_fullykiosk-0.0.11.dist-info

bash-5.1$ pip show python-fullykiosk
Name: python-fullykiosk
Version: 0.0.11
Summary: Python wrapper for Fully Kiosk Browser REST API
Home-page: https://github.com/cgarwood/python-fullykiosk
Author: Charles Garwood
Author-email: cgarwood@gmail.com
License: UNKNOWN
Location: /config/deps/lib/python3.10/site-packages
Requires:
Required-by:

[frenck]

Yay for building a custom non-root home-assistant image?

This project does not support that and thus is not something we consider.

[0dragosh]

Yay for building a custom non-root home-assistant image?

This project does not support that and thus is not something we consider.

That's mighty decisive and fast for something as important as a non-root container. Care to add some arguments or do you just like repeating yourself in all issues?

[onedr0p]

Yay for building a custom non-root home-assistant image?

This project does not support that and thus is not something we consider.

Yes, I know. Maybe one day ™ Docker and containers in general should give you enough controls with with Container capabilities to have what you need without going full-root. But this a discussion for another place and time.

[frenck]

That's mighty decisive and fast for something as important as a non-root container. Care to add some arguments or do you just like repeating yourself in all issues?

I don't think this is the place to defend or argument that. These are design decisions and limitations this project has made and documented. If you want more context, you can check out the architecture repository and the documented ADRs in that repo.

Seeing the thumbs down is OK. You don't have to agree. I'm not saying I agree or disagree either. It's a decision for this project on what it supports and what not.

You are free to run Home Assistant any way you like, however, unsupported/out of scope setups are in the projects scope right now.

But this a discussion for another place and time.

Agree non of that is for this issue report.

[iamwillbar]

Some additional details from my experience:

• Fresh install of Home Assistant using the k8s-at-home Helm chart and the 2022.7 homeassistant/home-assistant Docker image -> unable to launch configuration flows for multiple custom components due to them not being able to find dependencies.
• Downgrade that installation to the 2022.6.7 image -> same custom components can launch configuration flows.

[andjos]

I'm running on Kubernetes (not rootless), workaround by making HA think it's running as a container. (add file /.dockerenv) It still not installs dependencies to /config/deps but works..

[muhlba91]

@frenck what‘s the current plan/idea to get this fixed?

It seems to affect quite some users for whom it‘s now impossible to upgrade HA without breaking components, and possibly also automations, etc. Hence, it would be amazing seeing a fix coming rather soon-ish still to the 2022.7.x release. If the official communication is to stay on 2022.6.x (or apply a work-around for now?), I’d very much like to get an idea of the current plan and ETA.

I do understand the concerns mentioned about just removing this line due to some older issue with certain distros, though. Is this line still necessary for the Python 3.10 upgrade, or can you distinguish between those distros and other (installation methods)?

Please let us know if you need additional support and/or whether some of us can help.

[frenck]

what‘s the current plan/idea to get this fixed?

A fix was just merged. In general, this is an open source project. Plans are made by contributions.

[muhlba91]

Awesome, thanks for the fix!

Yes, sure. That's why I asked if support is needed here as I wasn't aware if you looked into it already and whether just removing it is safe due to the older issue mentioned.