search

internetarchive / warcprox

WARC writing MITM HTTP/S proxy
381 stars 54 forks source link

updates for Python 3.11 #192

Closed galgeek closed 1 year ago

vbanos commented 1 year ago

If we are dropping support for python <3.8, we can make many more improvements. E.g. from __future__ import absolute_import is not needed any more.

Also, all class definitions that are like class PlaybackIndexDb(object) could be become class PlaybackIndexDb

Finally, we'll be able to use f-strings which improve readability a lot (that's a big change, I don't suggest doing it now).

vbanos commented 1 year ago

What is the python version used when you run tests/run-all-tests.sh I see libpython2.7-dev libpython3-dev and a very old pip version pip-9.0.1.tar.gz in https://github.com/internetarchive/warcprox/blob/Py311/tests/Dockerfile Maybe you need to update this as well.

vbanos commented 1 year ago

The benchmarks use a very old version of aiohttp. https://github.com/internetarchive/warcprox/blob/Py311/benchmarks/requirements.txt Does it work with py311 ? Maybe you can update it as well https://pypi.org/project/aiohttp/#history

vbanos commented 1 year ago

I got this branch in my computer, I installed using pip install . in my existing py38 venv and got the following error when trying to run it. Thank you.

/opt/spn/lib/python3.8/site-packages/rethinkdb-2.3.0.post6-py3.8.egg/rethinkdb/net.py:312: SyntaxWarning: "is not" with a literal. Did you mean "!="?
/opt/spn/lib/python3.8/site-packages/rethinkdb-2.3.0.post6-py3.8.egg/rethinkdb/net.py:312: SyntaxWarning: "is not" with a literal. Did you mean "!="?
Traceback (most recent call last):
  File "/opt/spn/bin/warcprox", line 5, in <module>
    from warcprox.main import main
  File "/opt/spn/lib/python3.8/site-packages/warcprox/__init__.py", line 276, in <module>
    import warcprox.stats as stats
  File "/opt/spn/lib/python3.8/site-packages/warcprox/stats.py", line 32, in <module>
    from rethinkdb import RethinkDB; r = RethinkDB()
ImportError: cannot import name 'RethinkDB' from 'rethinkdb' (/opt/spn/lib/python3.8/site-packages/rethinkdb-2.3.0.post6-py3.8.egg/rethinkdb/__init__.py)
galgeek commented 1 year ago

It looks like pip failed to install the updated dependency, rethinkdb v.2.49, from doublethink.

I wonder if you have better luck creating a new virtualenv?

galgeek commented 1 year ago

A draft update to the benchmarks is in PR #193

galgeek commented 1 year ago

An earlier PR, #181, updates warcprox tests for github actions. It includes output from running against Python 3.11.2 that look good.

vbanos commented 1 year ago

I created a new virtualenv and pip install . worked. Thank you.