`python-matter-server` missing runtime dependencies in nixpkgs-unstable.

leonm1 commented 1 year ago

Describe the bug

I would like to run home-assistant with the matter integration. I am unable to do so, since python-matter-server's matter-server fails with the following error due to missing dependencies:

$ env NIXPKGS_ALLOW_INSECURE=1 nix shell nixpkgs#python-matter-server --impure
$ matter-server
Traceback (most recent call last):
  File "/nix/store/pjabn599v4wksg4gfl5zwr659bpz88ch-python3.10-python-matter-server-3.7.0/bin/.matter-server-wrapped", line 6, in <module>
    from matter_server.server.__main__ import main
  File "/nix/store/pjabn599v4wksg4gfl5zwr659bpz88ch-python3.10-python-matter-server-3.7.0/lib/python3.10/site-packages/matter_server/server/__init__.py", line 3, in <module>
    from .server import MatterServer
  File "/nix/store/pjabn599v4wksg4gfl5zwr659bpz88ch-python3.10-python-matter-server-3.7.0/lib/python3.10/site-packages/matter_server/server/server.py", line 23, in <module>
    from ..server.client_handler import WebsocketClientHandler
  File "/nix/store/pjabn599v4wksg4gfl5zwr659bpz88ch-python3.10-python-matter-server-3.7.0/lib/python3.10/site-packages/matter_server/server/client_handler.py", line 12, in <module>
    from chip.exceptions import ChipStackError
ModuleNotFoundError: No module named 'chip.exceptions'

(Note that home-assistant-chip-core depends on openssl_1_1, which is end-of-life, and is another sign this may not be the right dependency for matter's matter integration)

Steps To Reproduce

Steps to reproduce the behavior:

With nixpkgs-unstable:

Run env NIXPKGS_ALLOW_INSECURE=1 nix shell nixpkgs#python-matter-server --impure
Run matter-server
See error message.

Confirmed on x86_64 and aarch64.

Expected behavior

I expect the matter server to run and expose an interface on port 5580.

Additional context

I did some investigation into the issue.

The package at issue appears to be home-assistant-chip-core, which indicates its homepage is https://github.com/home-assistant-libs/chip-wheels, which is just a collection of patches for https://github.com/project-chip/connectedhomeip. I tried applying these patches to a clean checkout of connectedhomeip, but it appears like all of them are included upstream as of the latest master
home-assistant-chip-core is simply fetched from PyPI, and the upstream package is lacking the same files.
I think the solution is to build the Python controller in the connectedhomeip monorepo, but my knowledge of this project is limited, and it appears to use some non-nix-y things in its build script (like fetching its own dependencies from PyPI).
I also do not know what commit of connectedhomeip would be the right commit to build from.

Notify maintainers

@Mic92 @globin

(There's other maintainers "fab" and "hexa" listed in the home-assistant team, but neither of them appeared to be a part of the nix org)

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

I use flakes for my config. My current flake is built off of nixpkgs rev ace5093e36ab1e95cb9463863491bee90d5a4183

[user@system:~]$ nix run nixpkgs#nix-info -- -m
 - system: `"x86_64-linux"`
 - host os: `Linux 6.4.15, NixOS, 23.11 (Tapir), 23.11.20230915.ace5093`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - channels(matth): `"nixpkgs-22.11"`
 - channels(root): `""`
 - nixpkgs: `not found`

fabaff commented 1 year ago

Usernames in maintainer-list.nix do not always match GitHub's usernames. Notifying the whole team should probably work.

mweinelt commented 1 year ago

(Note that home-assistant-chip-core depends on openssl_1_1, which is end-of-life, and is another sign this may not be the right dependency for matter's matter integration)

It is the correct dependency, we're just talking about a shitty vendor consortium, where nobody cares about that kind of thing.

https://github.com/project-chip/connectedhomeip/issues/25688

The package at issue appears to be home-assistant-chip-core, which indicates its homepage is home-assistant-libs/chip-wheels, which is just a collection of patches for project-chip/connectedhomeip. I tried applying these patches to a clean checkout of connectedhomeip, but it appears like all of them are included upstream as of the latest master

We use the home assistant built wheels, because interacting with the upstream connectedhomeip repository was painful to me.

home-assistant-chip-core is simply fetched from PyPI, and the upstream package is lacking the same files.

It is in fact in there.

lib/python3.10/site-packages/chip/exceptions/__init__.py

I also do not know what commit of connectedhomeip would be the right commit to build from.

The same version as home assistant requires, if you want to use it with home assistant.

Please check https://github.com/NixOS/nixpkgs/pull/262521, where I included some fixes. It is not quite there yet, and I don't currently understand the problem, so some fresh eyes are welcome to take a look.

mweinelt commented 1 year ago

Also building things from the connectedhomeip is problematic, since the full source checkout is a whopping 27 GB, while our current output limit is 3 GB.

And then it goes on to download even more things to "bootstrap".

❯ ./scripts/build_python.sh
Input values: chip_detail_logging = false , chip_mdns = "", enable_pybindings = false, chip_case_retry_delta="", pregen_dir=""

  WELCOME TO...

         █
         █
     ▄   █   ▄                                █     █
     ▀▀█████▀▀      ▄▀▀▀▄ ▄▀▀▀▄    ▄▀▀▀▀▄█  ▀▀█▀▀▀▀▀█▀▀   ▄▀▀▀▀▄    ▄▀▀
   ▀█▄       ▄█▀   █     █     █  █      █    █     █    █▄▄▄▄▄▄█  █   
     ▀█▄   ▄█▀     █     █     █  █      █    █     █    █         █   
  ▄██▀▀█   █▀▀██▄  █     █     █   ▀▄▄▄▄▀█    ▀▄▄   ▀▄▄   ▀▄▄▄▄▀   █   
 ▀▀    █   █    ▀▀

  BOOTSTRAP! Bootstrap may take a few minutes; please be patient.

Downloading and installing packages into local source directory:

  Setting up CIPD package manager...[-]

SuperSandro2000 commented 10 months ago

Also building things from the connectedhomeip is problematic, since the full source checkout is a whopping 27 GB, while our current output limit is 3 GB.

I was double checking this and with submodules I got like 12 GB which is still insane. Also we are charting really deep into horrible enterprise software with horrible practices. So even if we could hack around this limit, fetch each submodule and then symlink them together, we would still need to fight the next steps already teasered.

mweinelt commented 10 months ago

Either way, this issue has been resolved a while ago.

nixos-discourse commented 4 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/compiling-the-matter-sdk-on-nixos/47742/1

NixOS / nixpkgs