pywb fails to load on macOS 11 Big Sur due to fakeredis issue

pywb fails to load on macOS 11.0 Big Sur due to an issue with fakeredis, if using Python <= 3.8. The issue in fakeredis is a problem with the find_library function from ctypes. See https://bugs.python.org/issue41179. The issue was patched in ctypes for Python 3.9 and 3.10 but will still affect projects using earlier versions of Python.

The code which calls find_library doesn't exist in the latest version of fakeredis, so updating the dependency would be one way to solve the problem.

Steps to reproduce the bug

Create an empty virtual environment using an affected version of Python. I used Python 3.6.9
Install pywb dependencies following steps in README
Try to run the wayback binary

It fails to start with the following error:

$ wayback
Traceback (most recent call last):
  File "/Users/robert/.virtualenvs/pywb/bin/wayback", line 33, in <module>
    sys.exit(load_entry_point('pywb==2.5.0', 'console_scripts', 'wayback')())
  File "/Users/robert/.virtualenvs/pywb/lib/python3.6/site-packages/pywb-2.5.0-py3.6.egg/pywb/apps/cli.py", line 20, in wayback
    desc='pywb Wayback Machine Server').run()
  File "/Users/robert/.virtualenvs/pywb/lib/python3.6/site-packages/pywb-2.5.0-py3.6.egg/pywb/apps/cli.py", line 89, in __init__
    self.application = self.load()
  File "/Users/robert/.virtualenvs/pywb/lib/python3.6/site-packages/pywb-2.5.0-py3.6.egg/pywb/apps/cli.py", line 178, in load
    from pywb.apps.frontendapp import FrontEndApp
  File "/Users/robert/.virtualenvs/pywb/lib/python3.6/site-packages/pywb-2.5.0-py3.6.egg/pywb/apps/frontendapp.py", line 26, in <module>
    from pywb.apps.rewriterapp import RewriterApp
  File "/Users/robert/.virtualenvs/pywb/lib/python3.6/site-packages/pywb-2.5.0-py3.6.egg/pywb/apps/rewriterapp.py", line 4, in <module>
    from fakeredis import FakeStrictRedis
  File "/Users/robert/.virtualenvs/pywb/lib/python3.6/site-packages/fakeredis-0.16.0-py3.6.egg/fakeredis.py", line 105, in <module>
    raise ImportError('fakeredis: unable to find libc or equivalent')
ImportError: fakeredis: unable to find libc or equivalent

A local workaround is to change the code just above where the error is produced in fakeredis.py from:

_libc_library = find_library('c') or find_library('msvcrt') or find_library('System')

To:

_libc_library = 'libc.dylib'

Environment

macOS 11.2 Big Sur
Python 3.6.9 installed via pyenv

Additional context

The issue was originally encountered in the context of a new version of Hypothesis Via proxy service. We currently use Python 3.6.9 for all of our projects, though we plan to update in future as Python 3.6 and 3.7 are in maintenance mode. Python 3.9 was still only released very recently (Oct 2020), so updating to it may be difficult for some projects.

For anyone else coming across this, we implemented a local workaround in our application that uses pywb:

import ctypes.util
import sys
from ctypes import CDLL

def patch_ctypes_on_macos_11():  # pragma: no cover
    """Work around a problem loading pywb under macOS 11.
    pywb has a dependency on an old version of fakeredis which fails to load
    when using Python <= 3.8 under macOS 11.  This function patches `ctypes.util.find_library`
    to work around the issue.
    """

    if sys.platform != "darwin":
        return

    def _find_library_patched(name):
        path = f"lib{name}.dylib"
        try:
            # In macOS 11, some system libraries don't exist on disk. Instead you test
            # the validity of a library path by trying to load it.
            # See https://bugs.python.org/issue41179.
            CDLL(path)
        except OSError:
            # Fall back to the un-patched version.
            path = ctypes.util.find_library(name)
        return path

    ctypes.util.find_library = _find_library_patched

# Apply ctypes patch before pywb is imported.
patch_ctypes_on_macos_11()

webrecorder / pywb