health: scrapydweb fails to launch, seems to require newer version

[pdurbin]

As I first reported at https://github.com/mlcommons/croissant/issues/530#issuecomment-2096806017 I tried to follow the README under /heath but scrapydweb failed to launch. err.txt is the full error but some highlights include:

• RuntimeError: Working outside of application context.
• sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: metadata
• (Background on this error at: https://sqlalche.me/e/20/e3q8)

@JR-1991 let me know he's getting the same error, which he believes is related to this issue:

• https://github.com/my8100/scrapydweb/issues/205

He said, "Scrapydweb is 1.5.0 on PyPI but coming from the requirements.txt it installs 1.4.0. I guess due to other dependencies using it already."

He suggested installing the dev version like this and it worked for me:

python -m pip install git+https://github.com/my8100/scrapydweb.git

Here's my the output from pip freeze on my machine before I installed the dev version:

absl-py==2.1.0
APScheduler==3.10.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
attrs==23.2.0
Automat==22.10.0
blinker==1.8.2
Brotli==1.1.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
constantly==23.10.4
cryptography==42.0.7
cssselect==1.2.0
decorator==5.1.1
duckdb==0.10.2
etils==1.8.0
filelock==3.14.0
Flask==3.0.3
Flask-Compress==1.15
Flask-SQLAlchemy==3.1.1
fsspec==2024.3.1
greenlet==3.0.3
huggingface-hub==0.23.0
hyperlink==21.0.0
idna==3.7
importlib_resources==6.4.0
incremental==22.10.0
isodate==0.6.1
itemadapter==0.8.0
itemloaders==1.2.0
itsdangerous==2.2.0
Jinja2==3.1.4
jmespath==1.0.1
joblib==1.4.2
jsonpath-rw==1.4.0
liac-arff==2.5.0
logparser==0.8.2
lxml==5.2.1
MarkupSafe==2.1.5
minio==7.2.7
mlcroissant==1.0.5
networkx==3.3
numpy==1.26.4
openml==0.14.2
packaging==24.0
pandas==2.2.2
pandas-stubs==2.2.1.240316
parsel==1.9.1
pexpect==4.9.0
ply==3.11
polars==0.20.23
Protego==0.3.1
ptyprocess==0.7.0
pyarrow==16.0.0
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycparser==2.22
pycryptodome==3.20.0
PyDispatcher==2.0.7
pyOpenSSL==24.1.0
pyparsing==3.1.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
queuelib==1.7.0
rdflib==7.0.0
requests==2.31.0
requests-file==2.0.0
scikit-learn==1.4.2
scipy==1.13.0
Scrapy==2.11.1
scrapyd==1.4.3
scrapyd-client==1.2.3
scrapydweb==1.4.0
service-identity==24.1.0
setuptools==69.5.1
six==1.16.0
SQLAlchemy==2.0.30
threadpoolctl==3.5.0
tldextract==5.1.2
tqdm==4.66.4
Twisted==24.3.0
types-pytz==2024.1.0.20240417
typing_extensions==4.11.0
tzdata==2024.1
tzlocal==5.2
uberegg==0.1.1
urllib3==2.2.1
w3lib==2.1.2
Werkzeug==3.0.3
xmltodict==0.13.0
zipp==3.18.1
zope.interface==6.3
zstandard==0.22.0

This is the version of scrapydweb I'm using now:

scrapydweb @ git+https://github.com/my8100/scrapydweb.git@8de7ede6fe317e3ef1688572c5a0e8112cc3232f

[marcenacp]

Hi @pdurbin, just saw this issue.

You said you forced the installation from the source (at 8de7ede6fe317e3ef1688572c5a0e8112cc3232f it seems the version is 1.5.0), yet from your pip freeze the installed version is still scrapydweb==1.4.0. Would fixing scrapydweb>=1.5.0 in the requirements.txt be sufficient? Do you have a PR?

Thanks!

[JR-1991]

@marcenacp - This will likely fix it. I suspect another package in the list has a scrapydweb==1.4.0 dependency, which means that the scrapydweb from the requirements is skipped since no version is specified. Happy to open a PR 😊

Just tested on a fresh conda env and the resulting version is 1.4.0

[pdurbin]

Would fixing scrapydweb>=1.5.0 in the requirements.txt be sufficient?

I tried this but pip install is taking forever and I'm getting messages like this:

INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

It seems to be in a loop. 🤔

And sorry for my confusing message above. I was trying to show before and after. The long pip freeze output was from just following the readme. The other line was just showing the "after", the version I got after running python -m pip install git+https://github.com/my8100/scrapydweb.git.