AttributeError exception causing API 500 errors being experienced when searching Dashboards

Ralnoc commented 4 years ago

Issue Summary

In some situations, when doing a search for dashboards is resulting in an AttributeError exception on obj.user.to_dict(). I have been unable to yet pinpoint any direct cause, only the process that leads up to the issue. In addition, the issue only manifests when searching, and only with some characters, not all. Primarily if there is a large number of results. If I just navigate through the pages on the Dashboards list, there is no issues experienced.

The order that I have been able to trace through the code, which leads to this problem is this:

User types in search in the Dashboards page of Re:Dash DashboardListResource is called to search DB for dashboards the meets the query using models.Dashboard.search - Successful DashboardListResource orders the results of that query - Successful DashboardListResource calls serialize_dashboard as part of the pagination process for the the results of the order_results - Successful handlers.base.paginate paginates the results and passes it over to serialize_dashboards - Successful serialize_dashboard executes to serialize the paginated results to return - Fails ( items = [serializer(result) for result in results.items] fails with the exception AttributeError: 'NoneType' object has no attribute 'to_dict')

This issue appears to have started after we upgraded from Re:Dash 6.0.0 to 8.0.0. We used the normal DB upgrade scripts as part of the upgrade process. We have tested the Queries and Users API endpoints and none of them throw this exception. It only happens on Dashboards and only when doing a search.

We suspect it is possible an issue with SQLAlchemy related to: https://github.com/sqlalchemy/sqlalchemy/issues/3650

This is the traceback being experienced:

[2020-04-14 12:10:05,461] ERROR in app: Exception on /api/dashboards [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 477, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/flask_login/utils.py", line 228, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/app/redash/handlers/base.py", line 31, in dispatch_request
    return super(BaseResource, self).dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/flask_restful/__init__.py\", line 587, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/app/redash/permissions.py", line 67, in decorated
    return fn(*args, **kwargs)
  File "/app/redash/handlers/dashboards.py", line 76, in get
    serializer=serialize_dashboard,
  File "/app/redash/handlers/base.py", line 108, in paginate
    items = [serializer(result) for result in results.items]
  File "/app/redash/serializers/__init__.py", line 214, in serialize_dashboard
    'user': obj.user.to_dict(),
AttributeError: 'NoneType' object has no attribute 'to_dict'

Steps to Reproduce

Have a number of Dashboards with content in them/ My testing was done with 12 to 15.
Set the Page Size (Results count) to 5 on the Dashboards page
Enter a one letter search that matches most of your results.
It will either immediately throw the exception with the API returning a 500, or if you click on the next page it will.

This is a breaking behavior the impairs the ability to Navigate Dashboards.

Technical details:

Redash Version: 8.0.0+b32245
Browser/OS: Chrome (Running on Linux, impacts all users, regardless of OS or Browser)
How did you install Redash: Docker

Ralnoc commented 4 years ago

I have made an alteration that on Initial testing seems to resolve the issue, and so far does not show any adverse effects having this change in place. This makes sense as it is likely related to the issue I linked in the original description which revolves entirely around the use of the subqueryload() function.

Making the following alteration in redash/models/__init__.py has so far stopped the manifestation of this issue in our lab environment.

diff --git a/redash/models/__init__.py b/redash/models/__init__.py
index cc16088..85c45ae 100644
--- a/redash/models/__init__.py
+++ b/redash/models/__init__.py
@@ -883,7 +883,7 @@ class Dashboard(ChangeTrackingMixin, TimestampMixin, BelongsToOrgMixin, db.Model
         query = (
             Dashboard.query
             .options(
-                subqueryload(Dashboard.user).load_only('_profile_image_url', 'name'),
+                joinedload(Dashboard.user).load_only('_profile_image_url', 'name'),
             )
             .outerjoin(Widget)
             .outerjoin(Visualization)

I have not identified any performance issues with this change yet. We will be rolling this change into Dev today as a means to validate the behavior. If it continues to resolve the issue, we should have it running in Production by end of day tomorrow and I will raise a PR here with the changes, referencing this issue.

Ralnoc commented 4 years ago

@susodapop - As you can see above, I've raised the PR to resolve the issue a minor reporting issue in the Presto Query Runner. I'll switch over there and monitor for any questions or concerns with the PR.

susodapop commented 4 years ago

I reproduced this bug on preview.

getredash / redash