pywb v-2.8.3: ModuleNotFoundError: No module named 'pyamf'

Hellseher commented 3 weeks ago

Describe the bug

pyamf is not listed in requirements.txt but requires during runtime. I've checked with GitHub search https://github.com/search?q=repo%3Awebrecorder%2Fpywb+pyamf&type=code

grep -r pyamf target/
target/pywb/rewrite/rewrite_amf.py:            from pyamf import remoting
target/pywb/warcserver/amf.py:from pyamf.remoting import Envelope, Request
target/pywb/warcserver/amf.py:from pyamf.flex.messaging import RemotingMessage
target/pywb/warcserver/inputrequest.py:from pyamf.remoting import decode
target/pywb/warcserver/test/test_amf.py:import pyamf
target/pywb/warcserver/test/test_amf.py:from pyamf.remoting import Envelope, Request, encode, decode
target/pywb/warcserver/test/test_amf.py:from pyamf.flex.messaging import RemotingMessage
target/pywb/warcserver/test/test_amf.py:pyamf.register_class(CustomObject, "custom.object")
target/pywb/warcserver/test/test_amf.py:    ev = Envelope(pyamf.AMF3)
target/pywb/warcserver/test/test_inputreq.py:from pyamf import AMF3
target/pywb/warcserver/test/test_inputreq.py:from pyamf.remoting import Request, Envelope, encode1

After inspecting setup.py: https://github.com/webrecorder/pywb/blob/97fffe3a345b10d1fb92f3a124aacfa010de9be7/setup.py#L62C1-L69C24

def load_requirements(filename):
    with open(filename, 'rt') as fh:
        requirements = fh.read().rstrip().split('\n')
    if sys.version_info > (3, 0):
        requirements.append("py3AMF")
    else:
        requirements.append("pyAMF")
    return requirements

Steps to reproduce the bug

git clone https://github.com/webrecorder/pywb

cd pywb

git checkout v-2.8.3

mkdir target

# Remove test requirements
rm -rf test_requirements.txt

# Merge all unique entries into one file.
sort -u *requirements* > r-all_tmp
rm -rf *requirements*
mv r-all_tmp requirements.txt

# Remove uwsgi wich comes as default and add gunicorn as an app server.
echo gunicorn >> requirements.txt
sed -i '/uwsgi/d' requirements.txt

# Install all dependencies into target
python3.11 -m pip install -r requirements.txt -t target
cp -R pywb target/

# Let Python find all modules
export PYTHONPATH=$PYTHONPATH:$PWD/target

# run app
./target/bin/gunicorn pywb.apps.wayback

Expected behavior

Application lists all requirements needed for runtime and starts up successfully when everything is installed.

Screenshots

./target/bin/gunicorn pywb.apps.wayback
[2024-11-05 10:57:07 +0000] [65409] [INFO] Starting gunicorn 23.0.0
[2024-11-05 10:57:07 +0000] [65409] [INFO] Listening at: http://127.0.0.1:8000 (65409)
[2024-11-05 10:57:07 +0000] [65409] [INFO] Using worker: sync
[2024-11-05 10:57:07 +0000] [65410] [INFO] Booting worker with pid: 65410
[2024-11-05 10:57:08 +0000] [65410] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "target/gunicorn/arbiter.py", line 608, in spawn_worker
    worker.init_process()
  File "target/gunicorn/workers/base.py", line 135, in init_process
    self.load_wsgi()
  File "target/gunicorn/workers/base.py", line 147, in load_wsgi
    self.wsgi = self.app.wsgi()
                ^^^^^^^^^^^^^^^
  File "target/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
                    ^^^^^^^^^^^
  File "target/gunicorn/app/wsgiapp.py", line 57, in load
    return self.load_wsgiapp()
           ^^^^^^^^^^^^^^^^^^^
  File "target/gunicorn/app/wsgiapp.py", line 47, in load_wsgiapp
    return util.import_app(self.app_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "target/gunicorn/util.py", line 370, in import_app
    mod = importlib.import_module(module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "target/pywb/apps/wayback.py", line 2, in <module>
    from pywb.apps.frontendapp import FrontEndApp
  File "target/pywb/apps/frontendapp.py", line 12, in <module>
    from pywb.recorder.recorderapp import RecorderApp
  File "target/pywb/recorder/recorderapp.py", line 15, in <module>
    from pywb.warcserver.inputrequest import DirectWSGIInputRequest
  File "target/pywb/warcserver/inputrequest.py", line 3, in <module>
    from pywb.warcserver.amf import Amf
  File "target/pywb/warcserver/amf.py", line 3, in <module>
    from pyamf.remoting import Envelope, Request
ModuleNotFoundError: No module named 'pyamf'
[2024-11-05 10:57:08 +0000] [65410] [INFO] Worker exiting (pid: 65410)
[2024-11-05 10:57:08 +0000] [65409] [ERROR] Worker (pid:65410) exited with code 3
[2024-11-05 10:57:08 +0000] [65409] [ERROR] Shutting down: Master
[2024-11-05 10:57:08 +0000] [65409] [ERROR] Reason: Worker failed to boot.

Environment

OS: Amazon Linux 2023.6.20241010

Browser N/A

python3.11 -m pip freeze
babel==2.16.0
boto3==1.35.54
botocore==1.35.54
brotlipy==0.7.0
certauth==1.3.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
cryptography==43.0.3
fakeredis==0.16.0
filelock==3.16.1
gevent==22.10.2
greenlet==2.0.2
gunicorn==23.0.0
idna==3.10
Jinja2==3.1.4
jmespath==1.0.1
lxml==5.3.0
MarkupSafe==3.0.2
packaging==24.1
portalocker==2.10.1
pycparser==2.22
pyOpenSSL==24.2.1
PySocks==1.7.1
python-dateutil==2.9.0.post0
PyYAML==6.0.2
redis==2.10.6
requests==2.32.3
requests-file==2.1.0
s3transfer==0.10.3
six==1.16.0
surt==0.3.1
tldextract==5.1.3
translate-toolkit==3.14.1
ua-parser==0.18.0
ujson==5.10.0
urllib3==2.2.3
warcio==1.7.4
wcwidth==0.2.13
webassets==2.0
webencodings==0.5.1
Werkzeug==2.2.3
wsgiprox==1.5.2
youtube-dl==2021.12.17
zope.event==5.0
zope.interface==7.1.1

Additional context

edsu commented 3 weeks ago

I'm confused, you saw that py3AMF is installed with setup.py for python3 but you didn't run it?

tw4l commented 3 weeks ago

Hi @Hellseher , now that we don't have support for Python 2 in pywb anymore, we could move the py3AMF requirement into requirements.txt. I'd be happy to review a PR if you'd like to submit one with the change, thanks.

Hellseher commented 3 weeks ago

@tw4l Please review when you have a chance https://github.com/webrecorder/pywb/pull/920

Hellseher commented 3 weeks ago

@edsu I saw it after investigation, my attempt was to use requirements.txt as the main source of truth for all dependencies.

tw4l commented 3 weeks ago

Fixed in #920

webrecorder / pywb