my8100 / scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
https://github.com/my8100/files
GNU General Public License v3.0
3.16k stars 565 forks source link

APScheduler 3.10 causing 500 errors #209

Closed bpmcdevitt closed 1 year ago

bpmcdevitt commented 1 year ago

Describe the bug: Python version 3.6. Using docker container python:3.6

To Reproduce: I had the following Python package versions installed:

root@de9470cdfefe:/opt/app# pip list  | grep -i -E 'flask|sql|aps'
APScheduler           3.10.0
Flask                 2.0.3
Flask-Compress        1.13
Flask-SQLAlchemy      2.5.1
SQLAlchemy            1.3.2

Stacktrace::

scrapydweb_1      | [2023-02-01 20:44:49,159] ERROR    in scrapydweb: Exception on / [GET]
scrapydweb_1      | Traceback (most recent call last):
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/selectable.py", line 2920, in __init__
scrapydweb_1      |     cols_present = bool(columns)
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 516, in __bool__
scrapydweb_1      |     raise TypeError("Boolean value of this clause is not defined")
scrapydweb_1      | TypeError: Boolean value of this clause is not defined
scrapydweb_1      | 
scrapydweb_1      | During handling of the above exception, another exception occurred:
scrapydweb_1      | 
scrapydweb_1      | Traceback (most recent call last):
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2073, in wsgi_app
scrapydweb_1      |     response = self.full_dispatch_request()
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1518, in full_dispatch_request
scrapydweb_1      |     rv = self.handle_user_exception(e)
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1516, in full_dispatch_request
scrapydweb_1      |     rv = self.dispatch_request()
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1502, in dispatch_request
scrapydweb_1      |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 83, in view
scrapydweb_1      |     self = view.view_class(*class_args, **class_kwargs)  # type: ignore
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/scrapydweb/views/index.py", line 10, in __init__
scrapydweb_1      |     super(IndexView, self).__init__()
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/scrapydweb/views/baseview.py", line 218, in __init__
scrapydweb_1      |     for job in self.scheduler.get_jobs(jobstore='default'))
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/apscheduler/schedulers/base.py", line 574, in get_jobs
scrapydweb_1      |     jobs.extend(store.get_all_jobs())
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/apscheduler/jobstores/sqlalchemy.py", line 89, in get_all_jobs
scrapydweb_1      |     jobs = self._get_jobs()
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/apscheduler/jobstores/sqlalchemy.py", line 141, in _get_jobs
scrapydweb_1      |     selectable = select(self.jobs_t.c.id, self.jobs_t.c.job_state).\
scrapydweb_1      |   File "<string>", line 2, in select
scrapydweb_1      |   File "<string>", line 2, in __init__
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
scrapydweb_1      |     return fn(*args, **kwargs)
scrapydweb_1      |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/selectable.py", line 2923, in __init__
scrapydweb_1      |     "columns argument to select() must "
scrapydweb_1      | sqlalchemy.exc.ArgumentError: columns argument to select() must be a Python list or other iterable

The fix: Downgrade APScheduler to 3.9.1

root@a7156c708554:/opt/app# pip list  | grep -i -E 'flask|sql|aps'
APScheduler           3.9.1
Flask                 2.0.3
Flask-Compress        1.13
Flask-SQLAlchemy      2.3.2
SQLAlchemy            1.3.2

After downgrading APScheduler no longer seeing the problem and the webserver can start up without 500 errors. I hope this can help somebody in the future trying to set this up.

my8100 commented 1 year ago

Please have a try for the latest scrapydweb v1.4.1, which supports Python 3.6 to 3.9

my8100 commented 1 year ago

Fixed in scrapydweb v1.4.1