Pip issues when installing Litex

[GuzTech]

I tried to run ./litex_setup.py --init --user -config=full on a fully updated Arch box and during the installation of Git repositories step, I get the following error:

./litex_setup.py --install --user --config=full
          __   _ __      _  __         
         / /  (_) /____ | |/_/         
        / /__/ / __/ -_)>  <           
       /____/_/\__/\__/_/|_|         
     Build your hardware, easily!      
          LiteX Setup utility.         

[   0.001] LiteX Setup auto-update...
[   0.403] LiteX Setup is up to date.
[   0.404] Installing Git repositories...
[   0.404] ------------------------------
[   0.404] Installing migen Git repository...
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.

    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Traceback (most recent call last):
  File "/home/oguz/Software/litex/./litex_setup.py", line 480, in <module>
    main()
  File "/home/oguz/Software/litex/./litex_setup.py", line 460, in main
    litex_setup_install_repos(config=args.config, user_mode=args.user)
  File "/home/oguz/Software/litex/./litex_setup.py", line 285, in litex_setup_install_repos
    subprocess.check_call("\"{python3}\" -m pip install --editable . {options}".format(
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '"/usr/bin/python3" -m pip install --editable . --user' returned non-zero exit status 1.

So apparently some things with pip have changed and the recommended way to install packages is to use pipx instead. I have modified the litex_setup.py script to use pipx but then I get the following issue:

/litex_setup.py --install --user --config=full
          __   _ __      _  __         
         / /  (_) /____ | |/_/         
        / /__/ / __/ -_)>  <           
       /____/_/\__/\__/_/|_|         
     Build your hardware, easily!      
          LiteX Setup utility.         

[   0.001] Installing Git repositories...
[   0.001] ------------------------------
[   0.001] Installing migen Git repository...

No apps associated with package migen or its dependencies. If you are attempting to install a library, pipx should not be used. Consider using pip or a similar tool instead.
Traceback (most recent call last):
  File "/home/oguz/Software/litex/./litex_setup.py", line 484, in <module>
    main()
  File "/home/oguz/Software/litex/./litex_setup.py", line 464, in main
    litex_setup_install_repos(config=args.config, user_mode=args.user)
  File "/home/oguz/Software/litex/./litex_setup.py", line 289, in litex_setup_install_repos
    subprocess.check_call("\"{python3}\" -m pipx install --editable . {options}".format(
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '"/usr/bin/python3" -m pipx install --editable . ' returned non-zero exit status 1.

I can actually install packages with pipx, but since (in this case) Migen is not an application, pipx returns with error code 1. So I have modified the script further to not check the return code and with that change it now also installs properly. The relevant hacky changes to litex_setup.py can be found here: https://github.com/GuzTech/litex/compare/master...pipx-patch

[trabucayre]

This behavior is more or less common to all distro since few month. One solution is to add --break-system-packages after pip install Another solution is to create a file $HOME/.config/pip/pip.conf with

[global]
break-system-packages = true

Second solution is great because is not limited to litex_setup.py but all users have to create this file, so maybe a small update to add argument looks better here.

[GuzTech]

How wise is it to potentially break system packages though? I do not know how realistic the odds are that some users will have issues if they use the break-system-packages method.

[trabucayre]

The main risk is to have for the same package one installed by the pkg manager and one by pip, with different version. Depending directory order with PYTHON_PATH or something similar, users may have issue due to this situation. I use a gentoo distro (where everything is built from sources), I have already observed issues due to a pkg installed by pip.

[sensille]

Same here, but when trying to fix litex after a system update. Instead of calling litex_setup installing meson via pacman fixed my install.

[AndrewD]

I found with arch linux that --user is no longer allowed, but --editable (development) installs are actually still installed per user so not specifying --user fixed this for me.

[josuah]

This sounds like the effect of https://peps.python.org/pep-0668/ which various distributions start slowly rolling out.

This pushes virtual environments down the throat of the user, but solves other problems for the user as well.

Not sure this can be helped at the level of LiteX.

Running python -m venv ~/litex/.venv then . ~/litex/.venv/bin/activate before ./litex_setup.py might be the fix.

If wanting to forget the fact that you use virtual environments, one way is to run python -m venv ~/.venv and add this to the ~/.profile:

if [ -f "$HOME/.venv/bin/activate" ]; then
    . "$HOME/.venv/bin/activate"
fi

Then every pip command would install things in ~/.venv instead of ~/.local, but besides than that, your system might work as it was before https://peps.python.org/pep-0668/ got enforced.

[pcotret]

This behavior is more or less common to all distro since few month. One solution is to add --break-system-packages after pip install Another solution is to create a file $HOME/.config/pip/pip.conf with

[global]
break-system-packages = true

Second solution is great because is not limited to litex_setup.py but all users have to create this file, so maybe a small update to add argument looks better here.

Got exactly the same issue on a Debian 12 distro. Second solution proposed by @trabucayre works in my case.