ValueError('Prefetch querysets cannot use values().')

[hanleyhansen]

Problem

When I upgraded to version 3.0.0 I started getting the following error ValueError('Prefetch querysets cannot use values().')

Here is the full traceback:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/.virtualenvs/tcj/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/.virtualenvs/tcj/lib/python2.7/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/.virtualenvs/tcj/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/.virtualenvs/tcj/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Projects/tcj/mainsite/management/commands/cron_hook.py", line 412, in handle
    getattr(CronFunctions, task_name)(*args)
  File "/Projects/tcj/mainsite/management/commands/cron_hook.py", line 119, in elasticsearch_reindex
    tcj_elasticsearch.reindex_listings(start_id=start_id, end_id=end_id)
  File "/Projects/tcj/TCJ/services/one_offs.py", line 452, in do_func
    return func(*args, **kwargs)
  File "/Projects/tcj/TCJ/tcj_elasticsearch.py", line 1602, in reindex_listings
    listings = BaseListing.listings.with_prefetches().using(db).indexable()\
  File "/Projects/tcj/business/models/listings.py", line 419, in with_prefetches
    ), to_attr='active_student_responses_prefetched'),
  File "/.virtualenvs/tcj/lib/python2.7/site-packages/django/db/models/query.py", line 1312, in __init__
    raise ValueError('Prefetch querysets cannot use values().')
ValueError: Prefetch querysets cannot use values().

I don't understand the error given this feature from almost a year ago.

Environment

• Django Model Utils version: 3.0.0
• Django version: 1.11.2
• Python version: 2.7.13

[hanleyhansen]

I noticed the following method on the Queryset class:

    def iterator(self):
        # Maintained for Django 1.8 compatability
        iter = super(InheritanceQuerySetMixin, self).iterator()
        if getattr(self, 'subclasses', False):
            extras = tuple(self.query.extra.keys())
            # sort the subclass names longest first,
            # so with 'a' and 'a__b' it goes as deep as possible
            subclasses = sorted(self.subclasses, key=len, reverse=True)
            for obj in iter:
                sub_obj = None
                for s in subclasses:
                    sub_obj = self._get_sub_obj_recurse(obj, s)
                    if sub_obj:
                        break
                if not sub_obj:
                    sub_obj = obj

                if getattr(self, '_annotated', False):
                    for k in self._annotated:
                        setattr(sub_obj, k, getattr(obj, k))

                for k in extras:
                    setattr(sub_obj, k, getattr(obj, k))

                yield sub_obj
        else:
            for obj in iter:
                yield obj

But as the docstring notes, it's for Django 1.8 compatibility.

The exception that gets caught in Django 1.11 is the following:

    if queryset is not None and not issubclass(queryset._iterable_class, ModelIterable):
            raise ValueError('Prefetch querysets cannot use values().')

Any thoughts?

[hanleyhansen]

I think I found the issue:

class InheritanceIterable(BaseIterable):
    def __iter__(self):
        queryset = self.queryset
        iter = ModelIterable(queryset)
        if getattr(queryset, 'subclasses', False):
            extras = tuple(queryset.query.extra.keys())
            # sort the subclass names longest first,
            # so with 'a' and 'a__b' it goes as deep as possible
            subclasses = sorted(queryset.subclasses, key=len, reverse=True)
            for obj in iter:
                sub_obj = None
                for s in subclasses:
                    sub_obj = queryset._get_sub_obj_recurse(obj, s)
                    if sub_obj:
                        break
                if not sub_obj:
                    sub_obj = obj

                if getattr(queryset, '_annotated', False):
                    for k in queryset._annotated:
                        setattr(sub_obj, k, getattr(obj, k))

                for k in extras:
                    setattr(sub_obj, k, getattr(obj, k))

                yield sub_obj
        else:
            for obj in iter:
                yield obj

InheritanceIterable should inherit from ModelIterable not BaseIterable. Hotpatching this works for me locally.

[eranrund]

@hanleyhansen what did you use to hotpatch the base class?

[hanleyhansen]

@eranrund I pointed to my commit in my fork of the project in my requirements.txt file so I installed my fixed version

[eranrund]

Ah :) Ended up doing the same. Thanks!

[jarshwah]

There is a patch https://github.com/jazzband/django-model-utils/pull/279/ that fixes this issue. Has been open quite awhile. Is someone able to merge please? If I can help in some way, please let me know :)

[hanleyhansen]

@jarshwah I wrote the patch 😄. Can't seem to get anybody to get eyes on it tho. Not sure where the contributors to this project are.

[jarshwah]

@hanleyhansen yep I know, just highlighting the PR against the issue a little more.

This is a jazzband project, which means there are a bunch of maintainers that contribute to various projects under the jazzband umbrella. It's my first time interacting with one though!

There are a bunch of issues and PRs open. Maybe helping out on some of the other items would help to encourage some attention on this particular issue 🤷‍♀️