Error when launching Storage Analytics manually

[ObadaS]

Hello,

Launching the Storage Analytics function manually results in an error that prevents it from running.

from analytics.tasks import create_storage_analytics_snapshot
eager_results = create_storage_analytics_snapshot.apply_async()

Inside docker-compose exec django ./manage.py shell_plus

Traceback (most recent call last):
  File "<input>", line 1, in <module>
    eager_results = create_storage_analytics_snapshot.apply_async()
  File "/usr/local/lib/python3.8/site-packages/celery/local.py", line 146, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/local/lib/python3.8/site-packages/celery/local.py", line 319, in _get_current_object
    return self.__evaluate__()
  File "/usr/local/lib/python3.8/site-packages/celery/local.py", line 349, in __evaluate__
    thing = Proxy._get_current_object(self)
  File "/usr/local/lib/python3.8/site-packages/celery/local.py", line 109, in _get_current_object
    return loc(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 434, in _task_from_fun
    base = base or self.Task
  File "/usr/local/lib/python3.8/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 1123, in Task
    return self.create_task_cls()
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 1002, in create_task_cls
    return self.subclass_with_self(
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 1026, in subclass_with_self
    Class = symbol_by_name(Class)
  File "/usr/local/lib/python3.8/site-packages/kombu/utils/imports.py", line 57, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "/usr/local/lib/python3.8/site-packages/bpython/curtsiesfrontend/repl.py", line 251, in load_module
    module = self.loader.load_module(name)
  File "<frozen importlib._bootstrap_external>", line 522, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1022, in load_module
  File "<frozen importlib._bootstrap_external>", line 847, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line 11, in <module>
    from celery import current_app, group, states
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 910, in _find_spec
  File "<frozen importlib._bootstrap>", line 884, in _find_spec_legacy
  File "/usr/local/lib/python3.8/site-packages/bpython/curtsiesfrontend/repl.py", line 280, in find_module
    loader = finder.find_module(fullname, path)
AttributeError: 'VendorImporter' object has no attribute 'find_module'

However, the weird part is that when the function is launched automatically, it seems to run successfully.

[2024-06-02 02:00:00,073: INFO/ForkPoolWorker-2] Task create_storage_analytics_snapshot started
[2024-06-02 02:02:35,970: INFO/Beat] Scheduler: Sending due task do_phase_migrations (competitions.tasks.do_phase_migrations)
[2024-06-02 02:02:35,981: INFO/MainProcess] Received task: competitions.tasks.do_phase_migrations[...]  
[2024-06-02 02:02:36,004: INFO/ForkPoolWorker-3] Checking 0 phases for phase migrations.
[2024-06-02 02:02:36,004: INFO/ForkPoolWorker-3] Task competitions.tasks.do_phase_migrations[...] succeeded in 0.022029880434274673s: None
[2024-06-02 02:07:36,093: INFO/Beat] Scheduler: Sending due task do_phase_migrations (competitions.tasks.do_phase_migrations)
[2024-06-02 02:07:36,104: INFO/MainProcess] Received task: competitions.tasks.do_phase_migrations[...]  
[2024-06-02 02:07:36,127: INFO/ForkPoolWorker-3] Checking 0 phases for phase migrations.
[2024-06-02 02:07:36,127: INFO/ForkPoolWorker-3] Task competitions.tasks.do_phase_migrations[...] succeeded in 0.021870918571949005s: None
[2024-06-02 02:10:08,097: INFO/ForkPoolWorker-2] Task create_storage_analytics_snapshot stoped. Duration = 53.388 seconds
[2024-06-02 02:10:08,100: INFO/ForkPoolWorker-2] Task analytics.tasks.create_storage_analytics_snapshot[...] succeeded in 608.0266336500645s: None

I am pretty sure that this error started appearing after #1459

[OhMaley]

So it looks like this error exists for all celery tasks that you want to starts manually. It might be related to the setuptools package version.

[OhMaley]

A good thing would be to upgrade the python packages. Lots of them are outdated

[Didayolo]

A good thing would be to upgrade the python packages. Lots of them are outdated

We're on it!

[ObadaS]

This issue has been fixed by https://github.com/codalab/codabench/pull/1543