can not install on Ubuntu 22.04 from git master

jeff350 commented 11 months ago

Description of Issue/Question

It appears that builds on Ubuntu 22.04 are broken for installs with git from master. It looks like this may be related to the changes to the requirements files since this is a Ubuntu system trying to install a windows only dependency. https://github.com/saltstack/salt/blob/master/requirements/base.txt#L32

from automated pipelines I can see this started failing between Nov 2nd and Nov 9th 2023. https://gitlab.com/saltstack-formulas/infrastructure/salt-image-builder/-/pipelines

Setup

(Please provide relevant configs (Be sure to remove sensitive info).) no setup needed

Steps to Reproduce Issue

(Include debug logs if possible, bootstrap-salt.sh -D.)

start a new ubuntu container docker container run -it ubuntu:jammy /bin/bash

root@e6e8f4b1dfb3:~# apt-get update
root@e6e8f4b1dfb3:~# apt-get install curl
root@e6e8f4b1dfb3:~# curl -L https://raw.githubusercontent.com/saltstack/salt-bootstrap/develop/bootstrap-salt.sh >bootstrap-salt.sh 
root@e6e8f4b1dfb3:~# bash bootstrap-salt.sh -XUdfPD -x python3 git master

relevant logs

 *  INFO: Running config_salt()
 * DEBUG: The passed destination(/etc/salt) is a directory
 * DEBUG: Full destination path is now: /etc/salt/minion
 * DEBUG: Moving /tmp/git/salt/conf//minion to /etc/salt/minion
 *  INFO: Running install_ubuntu_git()
 * DEBUG: __install_salt_from_repo_post_neon py_exe=python3
 * DEBUG: Installed pip version: pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Desired pip version '9.0.1' < Installed pip version '22.0.2'
 * DEBUG: Running 'pip3.10 install wheel setuptools>=9.1'
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (0.37.1)
Collecting wheel
  Downloading wheel-0.42.0-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.4/65.4 KB 1.3 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=9.1 in /usr/lib/python3/dist-packages (59.6.0)
Collecting setuptools>=9.1
  Downloading setuptools-69.0.2-py3-none-any.whl (819 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.5/819.5 KB 7.9 MB/s eta 0:00:00
Installing collected packages: wheel, setuptools
  Attempting uninstall: wheel
    Found existing installation: wheel 0.37.1
    Not uninstalling wheel at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'wheel'. No files were found to uninstall.
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed setuptools-69.0.2 wheel-0.42.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
 *  INFO: Installing salt using python3
 *  INFO: Downloading Salt Dependencies from PyPi
 * DEBUG: Running 'pip3.10 download -d /tmp/git/deps  .'
Processing /tmp/git/salt
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting wmi>=1.5.1
  Downloading WMI-1.5.1-py2.py3-none-any.whl (28 kB)
Collecting msgpack>=1.0.0
  Downloading msgpack-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (530 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 530.8/530.8 KB 5.3 MB/s eta 0:00:00
Collecting MarkupSafe
  Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting pymssql>=2.2.1
  Downloading pymssql-2.2.11-cp310-cp310-manylinux_2_28_x86_64.whl (4.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 36.1 MB/s eta 0:00:00
Collecting lxml>=4.6.3
  Downloading lxml-4.9.3-cp310-cp310-manylinux_2_28_x86_64.whl (7.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.9/7.9 MB 54.6 MB/s eta 0:00:00
Collecting rpm-vercmp
  Downloading rpm_vercmp-0.1.2.tar.gz (8.9 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting Jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 KB 10.4 MB/s eta 0:00:00
ERROR: Could not find a version that satisfies the requirement pywin32>=305 (from salt) (from versions: none)
ERROR: No matching distribution found for pywin32>=305
Failed to download salt dependencies
 *  INFO: Installing Downloaded Salt Dependencies
 * DEBUG: Running 'pip3.10 install --ignore-installed  /tmp/git/deps/*'
ERROR: Invalid requirement: '/tmp/git/deps/*'
Hint: It looks like a path. File '/tmp/git/deps/*' does not exist.
 * ERROR: Failed to run install_ubuntu_git()!!!
 * DEBUG: Cleaning up the Salt Temporary Git Repository
 * DEBUG: Removing the logging pipe /tmp/bootstrap-salt.logpipe
 * DEBUG: Removing the temporary apt error file /tmp/apt_error.RdSIkV

output can also be seen here https://gitlab.com/saltstack-formulas/infrastructure/salt-image-builder/-/jobs/5739273656

Versions and Systems

(salt --versions-report, bootstrap-salt.sh -v, system type and version, cloud/VM provider as appropriate.)

root@e6e8f4b1dfb3:/# bash bootstrap-salt.sh -v
bootstrap-salt.sh -- Version 2023.11.16

swohler commented 7 months ago

I am also seeing this same issue, attempting to install 3007.0 via pip on 22.04.4:

ERROR: Could not find a version that satisfies the requirement pywin32>=305 (from salt) (from versions: none)
ERROR: No matching distribution found for pywin32>=305

Versions:

Salt Version:
          Salt: 3007.0

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:34:22) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: ubuntu 22.04.4 jammy
        locale: utf-8
       machine: aarch64
       release: 6.6.8-gbb56f586effe
        system: Linux
       version: Ubuntu 22.04.4 jammy

salt-minion was installed with apt, as per the salt install instructions.

When I attempt to install the previous version 3006.8, there are issues with the version of pyzmq, even though the installed version already matches:

ERROR: Cannot install salt==3006.8 because these package versions have conflicting dependencies.

The conflict is caused by:
    salt 3006.8 depends on pyzmq==25.1.2
    salt 3006.8 depends on pyzmq>=20.0.0
    salt 3006.8 depends on pyzmq==25.0.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

pyzmq:

Name: pyzmq
Version: 25.1.2
Summary: Python bindings for 0MQ
Home-page: https://pyzmq.readthedocs.org
Author: Brian E. Granger, Min Ragan-Kelley
Author-email: zeromq-dev@lists.zeromq.org
License: LGPL+BSD
Location: /usr/local/lib/python3.10/dist-packages
Requires:
Required-by:

The issue is preventing another application from using the salt client [client = salt.client.Caller()], as I am unable to install the python package.

TimServers commented 6 months ago

I'm also seeing the same problem on Ubuntu 22.04


 *  INFO: Executed by: sh
 *  INFO: Command line: 'install_salt.sh -P -A xx.serxxxol.com.au -i cid-5xxx-xx-2xx5-or.sxxxxxl.com.au git v3007.0'

 *  INFO: System Information:
 *  INFO:   CPU:          GenuineIntel
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   5.15.0-105-generic
 *  INFO:   Distribution: Ubuntu 22.04

 *  INFO: Installing minion

 *  WARN: Post Neon git based installations will always install salt
 *  WARN: and its dependencies using pip which will be upgraded to
 *  WARN: at least v9.0.1, and, in case the setuptools version is also
 *  WARN: too old, it will be upgraded to at least v9.1

 *  WARN: You have 10 seconds to cancel and stop the bootstrap process...

 *  INFO: Found function install_ubuntu_git_deps
 *  INFO: Found function config_salt
 *  INFO: Found function preseed_master
 *  INFO: Found function install_ubuntu_git
 *  INFO: Found function install_ubuntu_git_post
 *  INFO: Found function install_ubuntu_restart_daemons
 *  INFO: Found function daemons_running
 *  INFO: Found function install_ubuntu_check_services
 *  INFO: Running install_ubuntu_git_deps()

Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20230311ubuntu0.22.04.1).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
 *  INFO: Git revision matches a Salt version tag, shallow cloning enabled.
 *  INFO: Attempting to shallow clone v3007.0 from Salt's repository https://github.com/saltstack/salt.git
Cloning into 'salt'...
Note: switching to '31c9d0df191009207c72ea73abfd3a1e3a0e6425'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

 *  INFO: Cloning Salt's git repository succeeded
Reading package lists...
Building dependency tree...
Reading state information...
g++ is already the newest version (4:11.2.0-1ubuntu1).
gcc is already the newest version (4:11.2.0-1ubuntu1).
python3-dev is already the newest version (3.10.6-1~22.04).
python3-setuptools is already the newest version (59.6.0-1.2ubuntu0.22.04.1).
python3-pip is already the newest version (22.0.2+dfsg-1ubuntu0.4).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
 *  INFO: Running config_salt()
 *  INFO: Running install_ubuntu_git()
Desired pip version '9.0.1' < Installed pip version '22.0.2'
Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (0.43.0)
Requirement already satisfied: setuptools>=9.1 in /usr/local/lib/python3.10/dist-packages (69.5.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#!/bin/bash
 *  INFO: Installing salt using python3
mkdir: cannot create directory '/tmp/git/deps': File exists
 *  INFO: Downloading Salt Dependencies from PyPi
Processing /tmp/git/salt
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting distro>=1.0.1
  Using cached distro-1.9.0-py3-none-any.whl (20 kB)
Collecting aiohttp>=3.9.0
  Using cached aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting wmi>=1.5.1
  Using cached WMI-1.5.1-py2.py3-none-any.whl (28 kB)
Collecting looseversion
  Using cached looseversion-1.3.0-py2.py3-none-any.whl (8.2 kB)
Collecting PyYAML
  Using cached PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
Collecting setproctitle>=1.2.3
  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)
Collecting rpm-vercmp
  Using cached rpm_vercmp-0.1.2.tar.gz (8.9 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting MarkupSafe
  Using cached MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting msgpack>=1.0.0
  Using cached msgpack-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (385 kB)
Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting jmespath
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting lxml>=4.6.3
  Using cached lxml-5.2.1-cp310-cp310-manylinux_2_28_x86_64.whl (5.0 MB)
Collecting importlib-metadata>=3.3.0
  Using cached importlib_metadata-7.1.0-py3-none-any.whl (24 kB)
ERROR: Could not find a version that satisfies the requirement pywin32>=305 (from salt) (from versions: none)
ERROR: No matching distribution found for pywin32>=305
Failed to download salt dependencies
 *  INFO: Installing Downloaded Salt Dependencies
ERROR: Invalid requirement: '/tmp/git/deps/*'
Hint: It looks like a path. File '/tmp/git/deps/*' does not exist.
 * ERROR: Failed to run install_ubuntu_git()!!!```

dmurphy18 commented 4 months ago

@jeff350 Closing this as fixed in release of the bootstrap 2024-07-12, see https://github.com/saltstack/salt-bootstrap/actions/workflows/ci.yml. Please feel free to reopen if other matters relevant to this issue come to light.

FredrikLiindgren commented 1 month ago

Also running into this issue when trying to install salt 3006.8 via pip 21.3.1, latest available pip version on python 3.6.8 (RHEL8). Was this fix only applied to packages findable by pip versions >=24.0?

dmurphy18 commented 1 month ago

@FredrikLiindgren PR testing includes testing on Rocky Linux 8, and git master worked there. Ubuntu 24.04 and I think Photon 5 had pip versions >=24.0 and had to adjust for that, otherwise PR testing passed.

The problem you may be running into is that 3006.x uses embedded Python 3.10 and that is what is tested against. Python 3.6 and 3.7 are long EOL, and not supported anymore, regardless of RedHat shipping with Python 3.6. That is why Salt went to the onedir architecture where current versions of Python and dependencies are embedded with Salt. It was a total PITA to try keep using OS supplied dependencies, esp. when they were so out of date and no longer maintained even before the OS EOL date, think RedHat 7, hardly any updates (even for CVEs) for 18 months before EOL.

If using Salt in production environments, then use packages, pip installing should be reserved for development and experimentation. And FYI RedHat 8 is basically EOL at this point, even though RedHat will keep it around for the 10 year mark.

Lastly, if this answer is not to your satisfaction, please open a new issue with details, rather than add on to a closed issue for a different OS, especially for one that is not even in the same family, that is, Debian/Ubuntu.

FredrikLiindgren commented 1 month ago

@dmurphy18 Thanks for your thorough reply.

My issue is when testing in a docker image in CI and installing dependencies for testing via the system distribution of pip.
I could probably just install salt via the rpm repository and install the testing dependencies in via salt-pip aswell.

Thanks again and apologies for posting in an unrelated issue.

saltstack / salt-bootstrap