search

Tijani-Dia / dj-tracker

A Django app that tracks your queries to help optimize them. Demo: https://dj-tracker-bakerydemo.fly.dev/dj-tracker/
https://tijani-dia.github.io/dj-tracker/
BSD 3-Clause "New" or "Revised" License
80 stars 3 forks source link

`AttributeError` after pickling queryset with related instances #25

Open jhonatan-lopes opened 1 year ago

jhonatan-lopes commented 1 year ago

When trying to request a URL after installing the package, I get an AttributeError when trying to access a page:

foundationmozillaorg-backend-1   | 17:12:17 web.1            | Internal Server Error: /en/privacynotincluded/
foundationmozillaorg-backend-1   | 17:12:17 web.1            | Traceback (most recent call last):
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 829, in _resolve_lookup
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     current = current[bit]
foundationmozillaorg-backend-1   | 17:12:17 web.1            | TypeError: 'ProductPage' object is not subscriptable
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | During handling of the above exception, another exception occurred:
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | Traceback (most recent call last):
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     rel_obj = self.field.get_cached_value(instance)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return instance._state.fields_cache[cache_name]
foundationmozillaorg-backend-1   | 17:12:17 web.1            | KeyError: 'image'
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | During handling of the above exception, another exception occurred:
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | Traceback (most recent call last):
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     response = get_response(request)
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/core/handlers/base.py", line 204, in _get_response
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     response = response.render()
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/response.py", line 105, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self.content = self.rendered_content
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/response.py", line 83, in rendered_content
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return template.render(context, self._request)
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/backends/django.py", line 61, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.template.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 170, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 38, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return compiled_parent._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 38, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return compiled_parent._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 38, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return compiled_parent._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     result = block.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     result = block.nodelist.render(context)
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/defaulttags.py", line 315, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/templatetags/cache.py", line 47, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     value = self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/defaulttags.py", line 214, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     nodelist.append(node.render_annotated(context))
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 90, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 195, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return template.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 172, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/wagtail/images/templatetags/wagtailimages_tags.py", line 101, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     image = self.image_expr.resolve(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 671, in resolve
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     obj = self.var.resolve(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 796, in resolve
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     value = self._resolve_lookup(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 837, in _resolve_lookup
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     current = getattr(current, bit)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/tracker.py", line 247, in wrapper
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     value = get_attr(instance, attr)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/field_descriptors.py", line 15, in __get__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.descriptor.__get__(instance, cls)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     rel_obj = self.get_object(instance)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return qs.get(self.field.get_reverse_related_filter(instance))
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/query.py", line 431, in get
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     num = len(clone)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self._fetch_all()
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self._result_cache = list(self._iterable_class(self))
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/tracker.py", line 117, in __iter__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     qs_tracker = QuerySetTracker(
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 370, in __init__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     instance_tracker.queryset.add_related_queryset(self)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 377, in add_related_queryset
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self.related_querysets.append(qs_tracker)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 330, in __getattr__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self.constructed.add(name)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 333, in __getattr__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     raise AttributeError(name)

To reproduce:

  1. Clone the Mozilla Foundation repo: https://github.com/MozillaFoundation/foundation.mozilla.org
  2. Install dj_tracker (adding either to dev-requirements.in or requirements.in)
  3. Update pip locks by running inv pip-compile-lock
  4. Run inv setup locally to setup the dev environment and create some fake data
  5. Navigate to the homepage ("localhost:8000") and see that it works fine
  6. Navigate to the *privacy not included page ("localhost:8000/en/privacynotincluded") and see the bug
Tijani-Dia commented 1 year ago

Hey @jhonatan-lopes, thanks for raising this. I spent some time debugging it and I found that it's related to how query trackers are pickled.

get_product_subset returns the products after they've been pickled via cache.get_or_set.

I recently implemented custom hooks to solve a similar issue: Fix pickling errors with related instances.

Currently, the __getstate__ method returns only values that were added to the tracker dictionary, not the one defined in the __slots__ attribute. That explains why we have the AttributeError: constructed since constructed is defined in the __slots__.

I thought about adding all the values in __slots__ when pickling but that would introduce more serious bugs if everything else remains unchanged. Handling stateful objects is quite tricky with pickle so I think we we'll need to think more about it.

When testing the view that raises errors, you can modify the last line from get_product_subset to the following as a temporary workaround:

return cache.get_or_set(key, products, 24 * 60 * 60)  # Set cache for 24h

# Change to

cache.get_or_set(key, products, 24 * 60 * 60)  # Set cache for 24h
return products
jhonatan-lopes commented 1 year ago

Hi @Tijani-Dia, thanks for the quick investigation! Your proposed solution worked like a charm!