Open joshuar opened 1 month ago
hacscred
commented
1 month ago What you need to do is for the Python process running HA to be able to import the module.
Home Assistant Core won't magically know how to do that for random code from the Internet, which HACS is.
As such, please close this issue and ask the author of the HACS module to document their work or read the manual, if you haven't done so.
There's nothing HA developers can or should do, IMO.
RosemaryOrchard
commented
1 month ago Home Assistant Core won't magically know how to do that for random code from the Internet, which HACS is.
That is incorrect, components installed through HACs and others can include that information in their manifest file. And this is evidenced by the information from OP about the logs referencing failing to install those modules.
hacscred
commented
1 month ago Home Assistant Core won't magically know how to do that for random code from the Internet, which HACS is.
That is incorrect, components installed through HACs and others can include that information in their manifest file. And this is evidenced by the information from OP about the logs referencing failing to install those modules.
I am sorry, but do I sound like the kind of person that needs assistance?
Obviously, if HACS doesn't do that, it won't work and even then there are a million reasons for why it wouldn't work in general, but thanks for wasting my time.
You have about the level of a LLM getting it wrong.
I kindly ask you to never speak to me again. Thank you.
Petro31
commented
1 month ago I kindly ask you to never speak to me again. Thank you.
You are anything but kind.
MasseR
commented
1 month ago I might have further datapoints for this. First of all, I'm new to home assistant, I started with 2024.09.something. I think this is relevant.
I installed the dirigera_platform during the 2024.09 era. Everything works
I have since upgraded to 2024.10.2. I'm now having issues with certain extensions
dirigera dependency was duplicated on the system. One in /config/deps/dirigera (v1.2.0) and one in /config/deps/libs/python3.12/site-packages/dirigera (v1.1.7). The former one had been updated, the latter one didn't. But the integration tried loading the latter.dreame-bot integration fails loading entirely due to missing dependencies. The first one to fail is the py-mini-racer. The dependency exists under deps but in /config/deps/py_mini_racer. The custom integration can't see this dependency
X-dark
commented
1 month ago @MasseR this is coherent with what I observed as well.
uv installs dependencies in /config/deps as we can see from HA startup logs. But custom integrations seems to look for dependencies elsewhere, hence the issue.
I can confirm symlinking /config/deps/lib/python3.12/site-packages to /config/deps fixes integration that were broken because of that (like mentioned in https://github.com/music-assistant/hass-music-assistant/issues/3031).
Not sure where to look next to understand where those two diverging paths come from.
danielgoepp
commented
1 month ago Hi @X-dark :) I see you found this thread too. I was just looking around to report the same, and it looks like you have already done so, thank you! I'm experiencing this also. Packages are getting installed to /config/deps, when HA seems to want to find them at /config/deps/lib/python3.12/site-packages. Subscribing to this thread to see if we get any updates or assistance here.
maxim-mityutko
commented
3 weeks ago Don't know where and how the HACS dependencies were installed originally (prior to 2024.10), currently they are located in /config/deps, which as per my understanding should be available in PATH for the libraries to be discoverable.
The sys.path is updated through the async_mount_local_lib_path method:
https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/bootstrap.py#L668-L676
Which in turn uses async_get_user_site to retrieve paths:
https://github.com/home-assistant/core/blob/06ea3a3014a0c2bdb162e723223e91489d90a48f/homeassistant/util/package.py#L133-L150
The problem however is that --user-site argument in args = [sys.executable, "-m", "site", "--user-site"] won't return the path applied in the PYTHONUSERBASE environment variable, this is facilitated by the --user-base key:
https://docs.python.org/3.12/library/site.html#site.USER_BASE
Hence the path does not end up in PATH. That's my understanding right now how the process works.
maxim-mityutko
commented
3 weeks ago Folks I did a bit more digging. There are 2 questions:
The reason why it's not in the PATH is mentioned in my post above.
On the question 1, this logic is influenced by the factors like, is HA is running in docker or venv or some other type of environment. I faced the issue with OpenPlantBook extension, and while it was working perfectly well for some folks, I faced the issue due to the fact that my dependencies were installed in /config/deps. While @slaxor505 and I are both running HA in contenerized environment, he uses docker and I use Kubernetes. And this makes a ton of difference, https://github.com/Olen/home-assistant-openplantbook/issues/32
The packages are installed with install_package method, there is a bit of conditional logic there that appends --target to the pip isntallation path:
https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/util/package.py#L124-L137
The method is called with a bunch of kwargs, that are generated within the pip_kwargs method:
https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/scripts/__init__.py#L54-L64
In pip_kwargs there is a conditional logic that adds target parameter and as I mentioned above, it use venv / docker criteria.
https://github.com/home-assistant/core/blob/487593af385fced4e15db84c8dbddc02c558ca23/homeassistant/requirements.py#L88-L97
For instance is_docker_env method: https://github.com/home-assistant/core/blob/cdf809926b01af7ae1c7595409be1e4c76ba9467/homeassistant/util/package.py#L30-L32
So with that being said, we need to figure what fix is feasible - adding proper catalogs to path or dealing with package installation target.
FL550
commented
3 weeks ago Thanks for digging this deep to troubleshoot the error.
Regarding your last question: I think we need to understand why the is_docker_env check was implemented for choosing the installation target. Just correcting the PATH might introduce problems we are currently not aware of.
FL550
commented
3 weeks ago I came to this issue, as many users of my integration were facing the same issues you describe with OpenPlantBook. If needed we could ask them which configuration they are using where the issue exists as well. From what I've already seen are many using HAOS.
maxim-mityutko
commented
3 weeks ago Yeah, it would be nice to understand which envs are susceptible to this problem.
Somehow I feel that this is related to the transition to uv and this PR: https://github.com/home-assistant/core/pull/125808
maxim-mityutko
commented
3 weeks ago @FL550 do you know how HAOS manages packages, is it venv?
CoolDuke
commented
3 weeks ago As a workaround I set PYTHONPATH to /config/deps in the container's environment so HASS can find and load the custom dependencies.
tbrasser
commented
3 weeks ago Running HA OS, same issue, is there a workaround available?
mots
commented
3 weeks ago @maxim-mityutko raises an interesting point with him running the container in kubernetes. I also ran into the issue and for me the container was running in podman.
It seems like the is_docker_env check should be updated so it works for other container environments.
mots
commented
3 weeks ago Doing some initial digging, to fix this issue for podman and kubernetes at least, the check should be extended to also check for
/run/containerenv (podman) and
/var/run/secrets/kubernetes.io (kubernetes)
mpreu
commented
2 weeks ago Doing some initial digging, to fix this issue for podman and kubernetes at least, the check should be extended to also check for
/run/containerenv(podman) and/var/run/secrets/kubernetes.io(kubernetes)
/var/run/secrets/kubernetes.io/... is not a fail-safe test for a Kubernetes environment. Using automountServiceAccountToken: false is considered a good practice when an application does not need the k8s API access. In this case the complete directory structure is not populated. IMHO it would be good to not "enforce" the automatic mount.
[var]/run/.containerenv is at least available for cri-o (https://github.com/cri-o/cri-o/issues/5461), but maybe not in other supported runtimes like containerd.
Maybe a fallback environment variable would be nice to have, as I can set it manually in each environment if needed.
EDIT:
Currently, the more viable test for a Kubernetes environment is using the default KUBERNETES_* environment variables (e.g. KUBERNETES_PORT, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_HOST). These injections cannot be disabled easily.
maxim-mityutko
commented
2 weeks ago @mpreu I agree, I think the environment variable should be used to indicate that HA is running in the containerized environment. More over, this environment variable can be included in the HA dockerfile by default, so you don't even need to set it on your own.
tbrasser
commented
2 weeks ago Just updated core to latest dev, frank_energie and openplantbook integrations now do load, now it's google_home not loading (possibly different issue)
maybe it's fixed in dev?
towo
commented
2 weeks ago Well, the only change after the previously mentioned PR is https://github.com/home-assistant/core/commit/59e5eb9a1c8cc37e757e9ccc0d05ccd3b9cf16c9, so that's the only one that could have fixed it. Essentially just adding a -m in there.
SphtKr
commented
5 days ago As a workaround I set
PYTHONPATHto/config/depsin the container's environment so HASS can find and load the custom dependencies.
+1 for this, worked for me... E.g.. in Kubernetes deployment YAML (since this seems to affect mostly k8s users):
spec:
containers:
- name: hass
image: docker.io/homeassistant/home-assistant:stable
env:
- name: "PYTHONPATH"
value: "/config/deps"Probably something similar if you're using Helm but I haven't gotten into that yet.
katter
commented
4 days ago As a workaround I set
PYTHONPATHto/config/depsin the container's environment so HASS can find and load the custom dependencies.+1 for this, worked for me... E.g.. in Kubernetes deployment YAML (since this seems to affect mostly k8s users):
spec: containers: - name: hass image: docker.io/homeassistant/home-assistant:stable env: - name: "PYTHONPATH" value: "/config/deps"Probably something similar if you're using Helm but I haven't gotten into that yet.
Thanks a lot @SphtKr !!!
Here a code snippet for those who have followed the gabe565 helm chart (e.g. via FluxCD):
values:
image:
repository: ghcr.io/home-assistant/home-assistant
tag: 2024.10.1
env:
TZ: "Europe/Berlin"
PYTHONPATH: "/config/deps"
devTechi
commented
4 days ago I am also facing the same issue with different integrations installed via HACS. I am also using a Kubernetes installation, to which I am about to migrate from a raspberry Pi - Docker Home Assistant. This issue held me from fully migrating.
I also came across setting PYTHONPATH, but setting it only to /config/deps didn't work and pip was complaining with some errors. Since dependencies get installed to /config/deps and also to /usr/local/lib/python3.12 (site-packages etc.) I had to set both locations. BUT the order is also important here. /usr/local/lib... needed to come first.
Here is my snippet from my StatefulSet (also usable for a Deployment):
containers:
- name: home-assistant
image: lscr.io/linuxserver/homeassistant:2024.11.1
env:
- name: PUID
value: "1000"
- name: PGID
value: "1000"
- name: TZ
value: Europe/Berlin
- name: PYTHONPATH
value: /usr/local/lib/python3.12:/config/deps # IMPORTANT PART (needs to be removed, when issue is fixed)Of course this is just a workaround as the Python version is fixed.
May be I just had to add both locations, because I didn't start from scratch and already had data/libs due to my migration.
x-real-ip
commented
4 days ago I am also facing the same issue with different integrations installed via HACS. I am also using a Kubernetes installation, to which I am about to migrate from a raspberry Pi - Docker Home Assistant. This issue held me from fully migrating.
I also came across setting
PYTHONPATH, but setting it only to/config/depsdidn't work and pip was complaining with some errors. Since dependencies get installed to/config/depsand also to/usr/local/lib/python3.12(site-packagesetc.) I had to set both locations. BUT the order is also important here./usr/local/lib...needed to come first.Here is my snippet from my StatefulSet (also usable for a Deployment):
containers: - name: home-assistant image: lscr.io/linuxserver/homeassistant:2024.11.1 env: - name: PUID value: "1000" - name: PGID value: "1000" - name: TZ value: Europe/Berlin - name: PYTHONPATH value: /usr/local/lib/python3.12:/config/deps # IMPORTANT PART (needs to be removed, when issue is fixed)Of course this is just a workaround as the Python version is fixed.
May be I just had to add both locations, because I didn't start from scratch and already had data/libs due to my migration.
Thanks for this one!
For those who are using Kustomize, here you can find a patch file for this issues.
rsteckler
commented
2 days ago Confirming that the devTechI/x-real-ip solution above worked for my kubernetes (RKE2) environment.
joshuar
commented
18 hours ago On 2024.11.2, using the official container, setting PYTHONPATH to /config/deps as an environment variables seems to lead to unintended glitches in the web interface:
TypeError: slugify() got an unexpected keyword argument 'separator') .However, I can install integrations like Meross and Tuya again.
The only reliable fix for me was https://github.com/home-assistant/core/issues/127966#issuecomment-2417022840
Herbs851
commented
8 hours ago Hi all,
I use Home assistant in a rootless podman container on debian 12.
herbs@polux:~#podman version
Client: Podman Engine
Version: 4.6.2
API Version: 4.6.2
Go Version: devel go1.22-2184a39477 Thu Dec 21 18:36:41 2023 +0000
Git Commit: 5db42e86862ef42c59304c38aa583732fd80f178
Built: Fri Dec 22 22:27:30 2023
OS/Arch: linux/amd64to correct this problem, taking the following https://github.com/home-assistant/core/issues/127966#issuecomment-2432185179 into account.
I created a .dockerenv file in the folder containing the rest of the HA config.
and finally I added the mount point to my systemd file.
I don't really like this solution, but for the moment I don't know what else to do.
With this kind of installation HA should check for the presence of the file /var/run/.containerenv.
The problem
Hey folks,
I’m in a bit of a pickle trying to add integrations from HACS.
I've recently started with a fresh install of 2024.10.1.
After successfully installing/downloading certain integrations through HACS, I am unable to add them to Home Assistant. When adding it, I get an error dialog with the message:
Config flow could not be loaded: {"message":"Invalid handler specified"}
In the Home Assistant log, it seems to correlate with a message like the following:
2024-10-08 21:40:31.362 INFO (SyncWorker_9) [homeassistant.util.package] Attempting install of meross_iot==0.4.7.3 2024-10-08 21:40:31.362 DEBUG (SyncWorker_9) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'meross_iot==0.4.7.3', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps'] 2024-10-08 21:40:40.992 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration meross_cloud: No module named 'meross_iot'
That’s specifically when trying to add the "Meross Integration" integration.
It seems integrations that need to install a Python module fail. Integrations that don’t need to install any Python modules, succeed, like themes.
For example, besides the "Meross Integration", the "Tapo Controller" integration also fails to get added, but the Python module in the ERROR log message is different.
Please let me know if you need any more information. As I can reproduce this in a container I can configure and retrieve whatever is needed.
Complete Logs:
home-assistant.tar.gz
Reproduction steps
Cheers!
What version of Home Assistant Core has the issue?
core-2024.10.1
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant Container
Integration causing the issue
No response
Link to integration documentation on our website
No response
Diagnostics information
Diagnostics dump
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
Reported to HACS https://github.com/hacs/integration/issues/4123 but was closed due to not being a HACS issue.