cve-search / CveXplore

CveXplore
https://cve-search.github.io/CveXplore/
GNU General Public License v3.0
33 stars 16 forks source link

Make Celery backend purely optional #295

Closed oh2fih closed 3 months ago

oh2fih commented 4 months ago

The documentation states that

The use of this backend is optional and CveXplore can function perfectly without one; the tasks it performs can well be handled by another mechanism. It is, however, a nice addition to the functionalities CveXplore offers.

However, the main.py imports

from CveXplore.core.celery_task_handler.task_handler import TaskHandler

that imports

from celery.schedules import schedule, crontab
from redbeat import RedBeatSchedulerEntry
from redbeat.decoder import RedBeatJSONEncoder
from redbeat.schedulers import ensure_conf

The problem is that even if you choose not to use the Celery runners you must have the celery[redis] and celery-redbeat dependencies installed. As the requirements are made modular and the celery is not included in the default, it is possible to install CveXplore without cvexplore[celery], causing a ModuleNotFoundError, e.g.:

Traceback (most recent call last):
  File "cve-search/sbin/db_updater.py", line 28, in <module>
    from CveXplore import CveXplore
  File "/usr/lib/python3/dist-packages/CveXplore/__init__.py", line 1, in <module>
    from CveXplore.main import CveXplore
  File "/usr/lib/python3/dist-packages/CveXplore/main.py", line 47, in <module>
    from CveXplore.core.celery_task_handler.task_handler import TaskHandler
  File "/usr/lib/python3/dist-packages/CveXplore/core/celery_task_handler/task_handler.py", line 12, in <module>
    from celery.schedules import schedule, crontab
ModuleNotFoundError: No module named 'celery'

Furthermore, the work on the celery runners have made CveXplore less modular, as requirements/modules/celery.txt inherits all-db.txt dependencies (MongoDB, MySQL, PostgreSQL & SQLite).

oh2fih commented 3 months ago

The problem in the title of this issue has been solved in #297.

I have opened issue #302 for the other problem mentioned to better describe the incomplete part.