[BUG] scripts/activate.sh fails on pigweed

[HendrikTBB]

Reproduction steps

Follow the instructions in BUILDING.md.

The 'activate' script gets to the 'setting up python environment' and then dies with a large spew of error messages:

Setting up CIPD package manager...done (43.1s) Setting up Project actions........skipped (0.1s) Setting up Python environment.....[-] ERROR at /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_build/facade.gni:187:7: Assertion failed. assert(_dep_is_in_link_dependencies, ^----- /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_log:impl must be listed in the pw_build_LINK_DEPS build arg when the /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_log:pw_log facade is in use. Please update your toolchain configuration. See /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_log/BUILD.gn:47:1: whence it was called. pw_facade("pw_log") { ^-------------------- See //src/lib/support/BUILD.gn:351:5: which caused the file to be included. "$dir_pw_log:impl", ^----------------- ['gn', 'gen', '/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/.environment/gn_out', '--args=chip_crypto="boringssl" dir_pigweed="/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo"']

Traceback (most recent call last): File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/virtualenv_setup/install.py", line 402, in install_packages subprocess.check_call( File "/usr/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['gn', 'gen', '/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/.environment/gn_out', '--args=chip_crypto="boringssl" dir_pigweed="/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo"']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 995, in sys.exit(main()) ^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 987, in main return EnvSetup(**vars(parse())).setup() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 577, in setup result = step(spin) ^^^^^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 766, in virtualenv if not virtualenv_setup.install( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/virtualenv_setup/install.py", line 443, in install install_packages(gn_target) File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/virtualenv_setup/install.py", line 410, in install_packages raise subprocess.CalledProcessError( subprocess.CalledProcessError: Command '['gn', 'gen', '/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/.environment/gn_out', '--args=chip_crypto="boringssl" dir_pigweed="/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo"']' returned non-zero exit status 1. Installing pip requirements for all... error: externally-managed-environment

× This environment is externally managed ?-> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.

Bug prevalence

Every time

GitHub hash of the SDK that was being used

9080cd3b

Platform

other

Platform Version(s)

Debian GNU/Linux trixie/sid

Anything else?

No response

[andy31415]

@HendrikTBB is this a clean checkout of the repository or are there modifications?

There are two errors: the link-deps (which I have no idea what is going on) and a complaint about virtualenv. We can try to fix at least the virtualenv - my understanding is that newer python does not like pip install on the system python, so if you do not have virtualenv, you cannot pip install virtualenv either.

How about these steps first:

• do you have virtualenv (which virtualenv)?
• assuming no based on the output, install it (maybe apt-get install virtualenv or apt-get install python3-virtualenv) and try again

If we manage to get rid of the virtualenv error, we can figure out things further.

[HendrikTBB]

I don't have a 'virtualenv' but I do have the 'python3-venv' Debian package installed. Also, the 'python3 -m venv' mechanism worked just fine for me installing a bunch of python packages required for the nRF52840 Thread firmware. But I have no idea how to convince the automated installation script to use that mechanism instead of whatever command it is actually running. I can't even tell what python package it is trying to install if it isn't the pigweed thing. So I can't just install it myself manually.

[HendrikTBB]

Okay, I've got the venv thing working. Not sure what happened before, maybe I had a typo in the path or something. But now adding the venv bin directory to the path is causing activate.sh to pick up my local pip and install the python stuff in there.

I still get the complaints about gn and pigweed not working, though.

And I can't move on to the next step of 'Build for the host' as that also requires running 'gn' which does not appear to exist'.

[HendrikTBB]

It seems likely that the missing 'gn' is part of the Debian generate-ninja package. I installed that.

It didn't help with the gn pigweed error of the activate script but it at least lets me run the 'gn gen out/host' command.

However, the ninja step then fails complaining about a missing 'zap-cli' and tells me to install the relevant relase from github/project-chip/zap/releases. Only problem is I already did that and all it provides is 'zap'. There is no 'zap-cli' program in the zap-linux-x64.deb package.

[HendrikTBB]

I created a symlink from zap to zap-cli and lets me run the build.

However, the checks (ninja -C out/host check) failed immediately with a lot of spew about pigweed ending with "No such file or directory: PosixPath('protoc')". I'm guessing that is related to pigweed not installing in the activate.sh script.

[andy31415]

Seems the same like #31851

Could you try:

git submodule update -f --init --recursive

for the submodule update and then try the bootstrap again?

[andy31415]

also see the note in the other bug:

check to see if there is a difference between pwd and realpath . in your environment. I was able to get past this issue by making those identical:
% cd $(realpath .)