Pywb fails to install on 3.11 & 3.12 (at least on M1 Macs)

[YousufSSyed]

This is the output I got when I did pyenv shell 3.11.2 and pip install pywb:

      ld: warning: -undefined dynamic_lookup may not work with chained fixups
      building 'gevent._gevent_c_greenlet_primitives' extension
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/yousuf/.pyenv/versions/3.11.2/include/python3.11 -I/Users/yousuf/.pyenv/versions/3.11.2/include/python3.11 -I/private/var/folders/55/p5p54zl95l123gy3xq4c68w00000gn/T/pip-install-muunm6ds/gevent_950a4868be3743bda7446af46329f5f0/deps -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/Users/yousuf/.pyenv/versions/3.11.2/include/python3.11 -c src/gevent/_greenlet_primitives.c -o build/temp.macosx-13.0-arm64-cpython-311/src/gevent/_greenlet_primitives.o -Wno-unreachable-code -Wno-deprecated-declarations -Wno-incompatible-sysroot -Wno-tautological-compare -Wno-implicit-function-declaration -Wno-unused-value -Wno-macro-redefined
      In file included from src/gevent/_greenlet_primitives.c:1080:
      /private/var/folders/55/p5p54zl95l123gy3xq4c68w00000gn/T/pip-install-muunm6ds/gevent_950a4868be3743bda7446af46329f5f0/deps/greenlet/greenlet.h:42:5: error: unknown type name 'CFrame'
          CFrame* cframe;
          ^
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for gevent
Failed to build gevent
ERROR: Could not build wheels for gevent, which is required to install pyproject.toml-based projects

[tw4l]

Thanks for reporting the issue. pywb currently has gevent pinned to 21.12.0, which doesn't have wheels for 3.11. From the gevent changelog it looks like newer releses of gevent have added preliminary support for Python 3.11 but may not have wheels for all platforms yet.

For now I'd recommend using Python 3.7-3.10, and we will keep this issue open to track support for 3.11.

[edsu]

I spent a bit of time trying to let gevent float to 3.1.2 and working through the effects of running pip install -r requirements.txt.

The good news is gevent v22.10.2 installs under Python3.11 now. But installing the latest werkzeug (which is unpinned) brings along the need to upgrade markupsafe and jinja2. I worked through the minimal changes to get that working, which mostly involved using pass_context instead of contextfunction. Once I'd done this wayback runs fine. You can see the changes here.

But then I ran the tests...

The bad news is that the tests install an old version of flask, which pulls in an old version of werkzeug which breaks things again. The reason why flask < v2 is installed is to get the old werkzeug which is needed by httpbin, which appears to be dead?

I think we could pin back werkzeug in our requirements.txt to solve this. But it seems like we would just be forestalling the inevitable?

httpbin isn't part of pywb proper, and is only needed for tests so maybe we should either:

a) find another solution to httpbin for the tests b) fork httpbin and apply the small change needed to get it working for our tests

It seems like b might be easier than a. But I'd be curious what others think.

[edsu]

It looks like there are quite a few other forks of httpbin out there with the Response fix applied. We could try using this one for our tests:

https://github.com/aaronhmiller/httpbin

And then try to wean ourselves off of using it?

[ikreymer]

Thanks for looking into this! Could you try that fork and see if it works in the short term, and then could move forward with that?

[edsu]

Using aaronhmiller/httpbin almost seems to work. Maybe some functionality isn't there any more, or the dependency updates introduced a problem in live rewriting? There are only 4 failures:

FAILED tests/test_live_rewriter.py::TestLiveRewriter::test_live_bad_content_length[frame] - webtest.app.AppError: Bad response: 400 Bad Request (not 200)
FAILED tests/test_live_rewriter.py::TestLiveRewriter::test_live_bad_content_length[non-frame] - webtest.app.AppError: Bad response: 400 Bad Request (not 200)
FAILED tests/test_live_rewriter.py::TestLiveRewriter::test_live_bad_content_length_with_range[frame] - webtest.app.AppError: Bad response: 400 Bad Request (not 206)
FAILED tests/test_live_rewriter.py::TestLiveRewriter::test_live_bad_content_length_with_range[non-frame] - webtest.app.AppError: Bad response: 400 Bad Request (not 206)