search

tosdr / edit.tosdr.org

šŸ‘šŸ‘Ž A new web app to rate services
https://edit.tosdr.org
GNU Affero General Public License v3.0
214 stars 37 forks source link

Problem deploying Hypothesis fork #1156

Closed michielbdejong closed 4 months ago

michielbdejong commented 4 months ago

I tried to follow the instructions in the readme of this repo but ran into this error in the Hypothesis make services step:

$ make services
____________________________________________________________________________________ summary _____________________________________________________________________________________
  dockercompose: commands succeeded
  congratulations :)
Traceback (most recent call last):
  File "/Users/michiel/gh/hypothesis/h/.tox/dockercompose/bin/pip-sync", line 5, in <module>
    from piptools.scripts.sync import cli
  File "/Users/michiel/gh/hypothesis/h/.tox/dockercompose/lib/python3.8/site-packages/piptools/scripts/sync.py", line 16, in <module>
    from .. import sync
  File "/Users/michiel/gh/hypothesis/h/.tox/dockercompose/lib/python3.8/site-packages/piptools/sync.py", line 11, in <module>
    from pip._internal.commands.freeze import DEV_PKGS
ImportError: cannot import name 'DEV_PKGS' from 'pip._internal.commands.freeze' (/Users/michiel/gh/hypothesis/h/.tox/dockercompose/lib/python3.8/site-packages/pip/_internal/commands/freeze.py)
ERROR: InvocationError for command /Users/michiel/gh/hypothesis/h/.tox/dockercompose/bin/pip-sync-faster requirements/dockercompose.txt --pip-args --disable-pip-version-check (exited with code 1)
____________________________________________________________________________________ summary _____________________________________________________________________________________
ERROR:   dockercompose: commands failed
make: *** [services] Error 1
michielbdejong commented 4 months ago

I'll create a bash script that automates the entire readme on a fresh Ubuntu 24.04 server and then we can debug more deterministically.

michielbdejong commented 4 months ago

Reproduced there (see https://github.com/tosdr/edit.tosdr.org/pull/1157 for the script)

michielbdejong commented 4 months ago

Looks like we need to do something like https://github.com/jazzband/pip-tools/pull/1906/files to support pip version 23. Let's see if there's an upstream fix in Hypothesis

michielbdejong commented 4 months ago

There is also a warning when running this as root:

WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

But on my Macbook I was running it as my own user, and this issue still happened, so this is probably not related.

michielbdejong commented 4 months ago

Trying to reproduce it in the main branch of upstream. For starters, that seems to use python 3.11.7 instead of 3.8.12. Let's see how far it gets.

michielbdejong commented 4 months ago

I don't see this error on the main branch of the upstream. I'll try rebasing our phoenix-integration branch on that, if that succeeds then that should fix it.

michielbdejong commented 4 months ago

Hm, that gives loads of conflicts around database auth, on which I don't have a lot of context. Maybe I'll try downgrading pip instead.

michielbdejong commented 4 months ago

make services calls:

tox -qe dockercompose --run-command 'sh -c "docker network create dbs 2>/dev/null || true"'
tox -qe dockercompose -- up -d

I see two options:

michielbdejong commented 4 months ago

Trying now with pyenv install 3.11.9

michielbdejong commented 4 months ago 
# make services
pyenv: tox: command not found

The `tox' command exists in these Python versions:
  3.8.12
  3.11.7

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
make: *** [Makefile:34: services] Error 127

So 3.11.7 it is, then

michielbdejong commented 4 months ago

python -m pip install -rrequirements/dockercompose.txt fails with:

root@do-staging:~/h# pip install --upgrade pip
Requirement already satisfied: pip in /root/.pyenv/versions/3.11.7/lib/python3.11/site-packages (23.2.1)
Collecting pip
  Obtaining dependency information for pip from https://files.pythonhosted.org/packages/8a/6a/19e9fe04fca059ccf770861c7d5721ab4c2aebc539889e97c7977528a53b/pip-24.0-py3-none-any.whl.metadata
  Downloading pip-24.0-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.0-py3-none-any.whl (2.1 MB)
   ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā” 2.1/2.1 MB 28.4 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.2.1
    Uninstalling pip-23.2.1:
      Successfully uninstalled pip-23.2.1
Successfully installed pip-24.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@do-staging:~/h# python -m pip install -rrequirements/dockercompose.txt
Collecting attrs==20.3.0 (from -r requirements/dockercompose.txt (line 7))
  Using cached attrs-20.3.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting bcrypt==4.0.1 (from -r requirements/dockercompose.txt (line 9))
  Using cached bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.metadata (9.0 kB)
Collecting build==0.8.0 (from -r requirements/dockercompose.txt (line 11))
  Using cached build-0.8.0-py3-none-any.whl.metadata (4.0 kB)
Collecting certifi==2022.12.7 (from -r requirements/dockercompose.txt (line 13))
  Using cached certifi-2022.12.7-py3-none-any.whl.metadata (2.9 kB)
Collecting cffi==1.15.1 (from -r requirements/dockercompose.txt (line 15))
  Using cached cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.1 kB)
Collecting chardet==4.0.0 (from -r requirements/dockercompose.txt (line 19))
  Using cached chardet-4.0.0-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting click==8.1.3 (from -r requirements/dockercompose.txt (line 21))
  Using cached click-8.1.3-py3-none-any.whl.metadata (3.2 kB)
Collecting cryptography==39.0.1 (from -r requirements/dockercompose.txt (line 23))
  Using cached cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.metadata (5.6 kB)
Collecting distro==1.5.0 (from -r requirements/dockercompose.txt (line 25))
  Using cached distro-1.5.0-py2.py3-none-any.whl.metadata (6.7 kB)
Collecting docker==5.0.0 (from docker[ssh]==5.0.0->-r requirements/dockercompose.txt (line 27))
  Using cached docker-5.0.0-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting docker-compose==1.29.2 (from -r requirements/dockercompose.txt (line 29))
  Using cached docker_compose-1.29.2-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting dockerpty==0.4.1 (from -r requirements/dockercompose.txt (line 31))
  Using cached dockerpty-0.4.1.tar.gz (13 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting docopt==0.6.2 (from -r requirements/dockercompose.txt (line 33))
  Using cached docopt-0.6.2.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting idna==2.10 (from -r requirements/dockercompose.txt (line 35))
  Using cached idna-2.10-py2.py3-none-any.whl.metadata (9.1 kB)
Collecting importlib-metadata==4.12.0 (from -r requirements/dockercompose.txt (line 37))
  Using cached importlib_metadata-4.12.0-py3-none-any.whl.metadata (4.0 kB)
Collecting jsonschema==3.2.0 (from -r requirements/dockercompose.txt (line 39))
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl.metadata (7.8 kB)
Collecting packaging==23.0 (from -r requirements/dockercompose.txt (line 41))
  Using cached packaging-23.0-py3-none-any.whl.metadata (3.1 kB)
Collecting paramiko==2.10.1 (from -r requirements/dockercompose.txt (line 43))
  Using cached paramiko-2.10.1-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting pep517==0.13.0 (from -r requirements/dockercompose.txt (line 45))
  Using cached pep517-0.13.0-py3-none-any.whl.metadata (3.8 kB)
Collecting pip-sync-faster==0.0.3 (from -r requirements/dockercompose.txt (line 47))
  Using cached pip_sync_faster-0.0.3-py3-none-any.whl.metadata (9.3 kB)
Collecting pip-tools==6.12.3 (from -r requirements/dockercompose.txt (line 49))
  Using cached pip_tools-6.12.3-py3-none-any.whl.metadata (23 kB)
Collecting pycparser==2.20 (from -r requirements/dockercompose.txt (line 53))
  Using cached pycparser-2.20-py2.py3-none-any.whl.metadata (907 bytes)
Collecting pynacl==1.4.0 (from -r requirements/dockercompose.txt (line 55))
  Using cached PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl.metadata (8.1 kB)
Collecting pyrsistent==0.17.3 (from -r requirements/dockercompose.txt (line 57))
  Using cached pyrsistent-0.17.3.tar.gz (106 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-dotenv==0.17.1 (from -r requirements/dockercompose.txt (line 59))
  Using cached python_dotenv-0.17.1-py2.py3-none-any.whl.metadata (17 kB)
Collecting pyyaml==5.4.1 (from -r requirements/dockercompose.txt (line 61))
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  Ɨ Getting requirements to build wheel did not run successfully.
  ā”‚ exit code: 1
  ā•°ā”€> [54 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Ɨ Getting requirements to build wheel did not run successfully.
ā”‚ exit code: 1
ā•°ā”€> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I'm a bit stuck on this path, will try removing tox now, see how that goes

michielbdejong commented 4 months ago

Ah sh*te, the same error shows up for make dev:

# make dev
Traceback (most recent call last):
  File "/root/h/.tox/dev/bin/pip-sync", line 5, in <module>
    from piptools.scripts.sync import cli
  File "/root/h/.tox/dev/lib/python3.11/site-packages/piptools/scripts/sync.py", line 16, in <module>
    from .. import sync
  File "/root/h/.tox/dev/lib/python3.11/site-packages/piptools/sync.py", line 11, in <module>
    from pip._internal.commands.freeze import DEV_PKGS
ImportError: cannot import name 'DEV_PKGS' from 'pip._internal.commands.freeze' (/root/h/.tox/dev/lib/python3.11/site-packages/pip/_internal/commands/freeze.py)
ERROR: InvocationError for command /root/h/.tox/dev/bin/pip-sync-faster requirements/dev.txt --pip-args --disable-pip-version-check (exited with code 1)

I'll see if I can edit that part of the makefile too, or alternatively I'll study the differences between our fork and the upstream, and see if I can use whatever combination of dependency versions they use that does work

michielbdejong commented 4 months ago

That worked! :)