Storage Analytics: Competitions and Users usage not loading (new issue)

[ObadaS]

Hello, this is a continuation of https://github.com/codalab/codabench/issues/1577 There is a new error when trying to open the tabs:

/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField StorageUsageHistory.at_date received a naive datetime (2024-08-31 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField CompetitionStorageDataPoint.at_date received a naive datetime (2024-08-01 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField CompetitionStorageDataPoint.at_date received a naive datetime (2024-08-31 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
Internal Server Error: /api/analytics/competitions_usage/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/decorators.py", line 54, in handler
    return func(*args, **kwargs)
  File "/app/src/apps/api/views/analytics.py", line 235, in competitions_usage
    username = su['competition__created_by__username'] or ("user #" + su['competition__created_by__id']) or "unknown user"
KeyError: 'competition__created_by__id'
Internal Server Error: /api/analytics/competitions_usage/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/decorators.py", line 54, in handler
    return func(*args, **kwargs)
  File "/app/src/apps/api/views/analytics.py", line 235, in competitions_usage
    username = su['competition__created_by__username'] or ("user #" + su['competition__created_by__id']) or "unknown user"
KeyError: 'competition__created_by__id'

[ObadaS]

Small update: The Users Usage tab works on the test server, but the competition Usage tab gives the same error

Also, this error is shown in Django, not the Site Worker (same for the error on the first comment)

/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField User.date_joined received a naive datetime (2024-01-01 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField Competition.created_when received a naive datetime (2024-01-01 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField Submission.created_when received a naive datetime (2024-01-01 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField StorageUsageHistory.at_date received a naive datetime (2024-10-13 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField StorageUsageHistory.at_date received a naive datetime (2024-11-14 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField CompetitionStorageDataPoint.at_date received a naive datetime (2024-10-13 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/usr/local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField CompetitionStorageDataPoint.at_date received a naive datetime (2024-11-14 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
Internal Server Error: /api/analytics/competitions_usage/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/decorators.py", line 54, in handler
    return func(*args, **kwargs)
  File "/app/src/apps/api/views/analytics.py", line 235, in competitions_usage
    username = su['competition__created_by__username'] or ("user #" + su['competition__created_by__id']) or "unknown user"
KeyError: 'competition__created_by__id'
Internal Server Error: /api/analytics/competitions_usage/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/decorators.py", line 54, in handler
    return func(*args, **kwargs)
  File "/app/src/apps/api/views/analytics.py", line 235, in competitions_usage
    username = su['competition__created_by__username'] or ("user #" + su['competition__created_by__id']) or "unknown user"
KeyError: 'competition__created_by__id'

[OhMaley]

After investigation the problem occurs when a competition is deleted. the CompetitionStorageDataPoint hold pointers to competitions, so if a competition gets deleted, then the information it holds all becomes null. In other worlds those CompetitionStorageDataPoint should be ignored in this API call as the size of the files it was measuring in the storage now belong to the orphan files category.

I'll come up with a fix shortly