Question about CHUNK_SIZE

[jonzlin95]

Just started using django polymorphic, and it's made life a lot easier so thanks! I have a view that serves a large number of objects (currently ~400), and noticed that the chunk size is causing a lot of extra queries to the DB.

try:
    from django.db.models.query import CHUNK_SIZE               # this is 100 for Django 1.1/1.2
except ImportError:
    # CHUNK_SIZE was removed in Django 1.6
    CHUNK_SIZE = 100

I'm wondering if there are any performance reasons for using 100 (and if anything bad will happen if I increase it), and also possibly moving this setting to a central settings.py configuration.

Cheers!

For reference the most relevant issue I could find: https://github.com/chrisglass/django_polymorphic/pull/35

[vdboor]

The mean reason is both historical (mimicking what Django does) and the fact that polymorphic breaks the queryset iteration a bit. After all, all base objects need to be fetched to determine which derived models should be read from the database.

I fully agree on making that a option, and making a full request is welcome for that!

My guess is that polymorphic still needs a chunk size, but it can be much larger. The resulting performance hit should be investigated, e.g. does reading 1000 base objects result in a better or worse performance? Also note that the WHERE id IN (..) has limits that need to be respected.

[nick-lehmann]

Are there any updates on this issue? Is it still planned to make chunking optionally by introducing, let's say, a parameter to disable chunking or setting the chunk size to another value?

[vdboor]

There aren't any plans; the main driving force for polymorphic is people who implement the features they need from it. Thus, I'm happy to accept pull requests that change this.

[yotamgod]

Is this issue meant to cause 10 queries to the database for every 1000 objects queried?