Error occurred loading flow for integration openplantbook: No module named 'openplantbook_sdk'

[mots]

Should this library ship with the component? I installed using HACS in a docker container.

[mots]

I ended up installing it via pip and it seems to work now, but this might still be an issue for other users?

[Olen]

It's in the manifest, so it should be installed automatically.

@slaxor505: Have you changed anything in your end?

https://github.com/Olen/home-assistant-openplantbook/blob/27a9fec72a8544e7e139a99fc669515f389adebd/custom_components/openplantbook/manifest.json#L20

[slaxor505]

The library is in PyPi. I have not changed anything since the release. It would be interesting to see HASS startup log when it is supposed to attempt and install the library automatically using the manifest. I'm wondering @mots if you have some old Python version because the library minimum is Python 3.9, however since you managed to install it via pip then it might be something else preventing its installation during HASS start up. Would you be able to send a log from HASS?

[maxim-mityutko]

I have the same error. HASS is deployed as container - home-assistant:2024.10.2 Upgraded the extension via HACS.

Logger: homeassistant.setup
Source: setup.py:334
First occurred: 13:48:24 (1 occurrences)
Last logged: 13:48:24

Setup failed for custom integration 'openplantbook': Unable to import component: No module named 'openplantbook_sdk'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
    self._component_future.result()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
    comp = await self.hass.async_add_import_executor_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/openplantbook/__init__.py", line 11, in <module>
    from openplantbook_sdk import MissingClientIdOrSecret, OpenPlantBookApi
ModuleNotFoundError: No module named 'openplantbook_sdk'

[slaxor505]

Hi Max. Can you provide HASS startup log please?

On Fri, 18 Oct 2024, 06:31 Max, @.***> wrote:

I have the same error. HASS is deployed as container - home-assistant:2024.10.2 Upgraded the extension via HACS.

Logger: homeassistant.setup Source: setup.py:334 First occurred: 13:48:24 (1 occurrences) Last logged: 13:48:24

Setup failed for custom integration 'openplantbook': Unable to import component: No module named 'openplantbook_sdk' Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component component = await integration.async_get_component() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component self._component_future.result() File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component comp = await self.hass.async_add_import_executor_job( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component ComponentProtocol, importlib.import_module(self.pkg_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func return func(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 995, in exec_module File "", line 488, in _call_with_frames_removed File "/config/custom_components/openplantbook/init.py", line 11, in from openplantbook_sdk import MissingClientIdOrSecret, OpenPlantBookApi ModuleNotFoundError: No module named 'openplantbook_sdk'

— Reply to this email directly, view it on GitHub https://github.com/Olen/home-assistant-openplantbook/issues/32#issuecomment-2420367338, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGJ4FDDUQOUMIV3UMOXN6TZ4AF7ZAVCNFSM6AAAAABQB5ZHQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRQGM3DOMZTHA . You are receiving this because you were mentioned.Message ID: @.***>

[maxim-mityutko]

hi @slaxor505 , below are the logs that I get after the container restart. (I've also tried to reinstall the extension for good measure) logs-from-home-assistant-in-home-assistant-5d86c4b867-gjwqg.log

[slaxor505]

Hi @maxim-mityutko. I'm a bit puzzled here as your startup log is very short. I don't see HASS is setting up components. What log level is set? For example my log much more talkative. Can you increase log level maybe? See attached.

If I remove the library from the python then I will see the following message where HASS attempts to install it. I'm wondering if it is some setting in HASS or not generic HASS container.

2024-10-19 10:23:16.637 INFO (SyncWorker_0) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7

hass.log

[slaxor505]

Hi all,

I was trying to replicate the issue but was unable to do so. I removed and downgraded to OPB 1.2 then I reset Docker container (hass config is persistent). Then I started brand the new container with OPB 1.2. I see that hass installs previous OPB integration library from Olen. Checked that integration works fine. Then upgraded to OPB 1.3, on startup HASS installed opb_sdk on startup with no issues. Then again I checked that integration works fine.

I'm quite puzzled why the issue occurs and need your help to troubleshoot it.

[maxim-mityutko]

Thanks for investigating @slaxor505, my logs were indeed set to WARNING. After I switched to INFO, I see:

2024-10-20 21:19:39.129 INFO (SyncWorker_2) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7

logs-from-home-assistant-in-home-assistant-5d86c4b867-dslsw.log

[maxim-mityutko]

And with DEBUG level (I only copy-pasted the relevant parts)...

2024-10-20 21:34:30.808 INFO (SyncWorker_7) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7
2024-10-20 21:34:30.808 DEBUG (SyncWorker_7) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'openplantbook-sdk==0.4.7', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']

So I see that pip installs the dependency in /config/deps, these are the contents:

 % ls -hla | grep openplant
drwxr-xr-x  2 root root   4 Oct 16 22:15 openplantbook_sdk
drwxr-xr-x  2 root root   9 Oct 16 22:15 openplantbook_sdk-0.4.7.dist-info

 % ls -hla openplantbook_sdk/
total 22K
drwxr-xr-x  2 root root   4 Oct 16 22:15 .
drwxr-xr-x 33 root root  35 Oct 16 22:15 ..
-rw-r--r--  1 root root 158 Oct 16 22:15 __init__.py
-rw-r--r--  1 root root 18K Oct 16 22:15 sdk.py

[Olen]

Didn't HA recently do some "smart" changes to the installation order of modules?

I don't have the docs in front of me here, but I seem to remember there was something with some of the other integrations as well, that needed a minor change to ensure the modules are installed in time.

[slaxor505]

I could not find anything like this in docs. https://developers.home-assistant.io/docs/creating_integration_manifest#requirements

I assume you are talking about other components in "dependencies". But it is not about python libraries but HASS components that the component is dependent on. https://developers.home-assistant.io/docs/creating_integration_manifest#dependencies

I'm wondering if it is specific to architecture. Can @maxim-mityutko let me know if you run it on Raspberry PI? I have not tested this on Arm architecture I can try to do so.

[slaxor505]

Thanks for the details @maxim-mityutko I dag deeper and I see that in my docker container config/deps directory is empty and all the python libs are installed in /usr/local/lib/python3.12/site-packages including openplantbook_sdk.

I have question, do you have any other python libs in "deps" directory too or it is only openplantbook_sdk in there? If it is the only library then does the directory actually visible to python interpreter? E.g. in %PATH% variable.

[maxim-mityutko]

It is not the only one:

But pretty much all libraries except for json_timeseries that are installed in this path are also present in /usr/local/lib/python3.12/site-packages

And the /config/deps is not in PATH:

kcm01:/usr/local/lib/python3.12/site-packages# echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[maxim-mityutko]

Seems like a related thread: https://github.com/home-assistant/core/issues/127966

[maxim-mityutko]

I'm wondering if it is specific to architecture. Can @maxim-mityutko let me know if you run it on Raspberry PI? I have not tested this on Arm architecture I can try to do so.

It's running on x86 node

[maxim-mityutko]

Don't know where and how the HACS dependencies were installed originally (prior to 2024.10). The sys.path is updated through the async_mount_local_lib_path method: https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/bootstrap.py#L668-L676

Which in turn uses async_get_user_site to retrieve paths: https://github.com/home-assistant/core/blob/06ea3a3014a0c2bdb162e723223e91489d90a48f/homeassistant/util/package.py#L133-L150

The problem however is that --user-site argument in args = [sys.executable, "-m", "site", "--user-site"] won't return the path applied in the PYTHONUSERBASE environment variable, this is facilitated by the --user-base key: https://docs.python.org/3.12/library/site.html#site.USER_BASE

Hence the path does not end up in PATH.

[slaxor505]

Good catch and it does seem that it is getting a known problem. I'm just curious how it works in my case. All libraries in my docker container have been installed in "site-packages" but not in "deps" which is empty. I also use x86 genuine HASS image. I'm pondering what it depends on and what could be a work around. One obvious is to issue "pip install openplantbook-sdk" within container but not really user friendly one.

[maxim-mityutko]

What do you see in the logs, is the --target specified?

�[36m2024-10-20 21:34:30.808 DEBUG (SyncWorker_7) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'openplantbook-sdk==0.4.7', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']�[0m

Installing manually can work as a temporaru workaround, but won't persist after container is recreated. So not really an option in the long run.

Do you by any chance have /.dockerenv in your container filesystem?

[slaxor505]

I reset the container and this is what I got from the startup logs.

2024-10-23 20:28:53.232 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7 2024-10-23 20:28:53.235 DEBUG (SyncWorker_1) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'openplantbook-sdk==0.4.7', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt']

I don't have --target parameter as you can see.

And I do have .dockerenv in the root

drwxr-xr-x 91 root root 4.0K Oct 23 20:28 . drwxr-xr-x 91 root root 4.0K Oct 23 20:28 .. -rwxr-xr-x 1 root root 0 Oct 23 20:28 .dockerenv -rw-r--r-- 1 root root 76 Oct 19 02:12 OFFICIAL_IMAGE drwxr-xr-x 2 root root 4.0K Jun 23 01:44 bin drwxr-xr-x 2 root root 12.0K Jun 22 22:24 command drwxr-xr-x 11 1024 users 4.0K Oct 23 20:28 config drwxr-xr-x 5 root root 360 Oct 23 20:28 dev drwxr-xr-x 41 root root 4.0K Oct 23 20:28 etc drwxr-xr-x 2 root root 4.0K Jun 19 00:16 home -rwxr-xr-x 1 root root 1.1K Oct 19 02:12 init drwxr-xr-x 15 root root 4.0K Jun 23 01:44 lib drwxr-xr-x 5 root root 4.0K Jun 19 00:16 media drwxr-xr-x 2 root root 4.0K Jun 19 00:16 mnt drwxr-xr-x 2 root root 4.0K Jun 19 00:16 opt drwxr-xr-x 6 root root 4.0K Nov 21 2023 package dr-xr-xr-x 303 root root 0 Oct 23 20:28 proc drwx------ 7 root root 4.0K Oct 23 20:36 root drwxr-xr-x 6 root root 4.0K Oct 23 20:28 run drwxr-xr-x 2 root root 4.0K Jun 23 01:44 sbin drwxr-xr-x 2 root root 4.0K Jun 19 00:16 srv dr-xr-xr-x 12 root root 0 Oct 23 20:28 sys drwxrwxrwt 2 root root 4.0K Oct 23 20:28 tmp drwxr-xr-x 57 root root 4.0K Oct 23 20:28 usr drwxr-xr-x 15 root root 4.0K Jun 23 01:44 var

[maxim-mityutko]

Ok, then I have a good idea of what's going on.

The packages are installed with install_package method: https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/util/package.py#L93

There is a bit of conditional logic there that appends --target to the pip isntallation path:

    if target:
        abs_target = os.path.abspath(target)
        args += ["--target", abs_target]
    elif (
        not is_virtual_env()
        and not (any(var in env for var in _UV_ENV_PYTHON_VARS))
        and (abs_target := site.getusersitepackages())
    ):
        # Pip compatibility
        # Uv has currently no support for --user
        # See https://github.com/astral-sh/uv/issues/2077
        # Using workaround to install to site-packages
        # https://github.com/astral-sh/uv/issues/2077#issuecomment-2150406001
        args += ["--python", sys.executable, "--target", abs_target]

It is called from here with kwargs, one of them is target. And here is how those kwargs are created: https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/requirements.py#L88-L97 And the interesting part is:

    if not (config_dir is None or pkg_util.is_virtual_env()) and not is_docker:
        kwargs["target"] = os.path.join(config_dir, "deps")

As you can see target parameter will be used and appended if all the conditions are met.

And the is_docker_env method: https://github.com/home-assistant/core/blob/cdf809926b01af7ae1c7595409be1e4c76ba9467/homeassistant/util/package.py#L30-L32

def is_docker_env() -> bool:
    """Return True if we run in a docker env."""
    return Path("/.dockerenv").exists()

I run my container in Kubernetes, i don't have this file, hence after all the chain of logic is applied I get the /config/deps as target. In your case, the target is not set during the instal_package call, and the elif logic also resolves to false, so the libraries are installed in the default path.