openedx / edx-django-utils

edX utilities for Django Application development.
https://edx.readthedocs.io/projects/edx-django-utils/en/latest/
Apache License 2.0
26 stars 20 forks source link

ENT-4236 | added chunked_queryset utility. #102

Closed HammadAhmadWaqas closed 3 years ago

HammadAhmadWaqas commented 3 years ago

Description:

  Added utity to Slice a queryset into chunks.

  The function slices a queryset into smaller QuerySets containing chunk_size objects and then yields them. It is
  used to avoid memory error when processing huge querysets, and also to avoid database errors due to the
  database pulling the whole table at once. Additionally, without using a chunked queryset, concurrent database
  modification while processing a large table might repeat or skip some entries.

JIRA:

ENT-4236

Dependencies:

List dependencies on other outstanding PRs, issues, etc.

Merge deadline:

List merge deadline (if any)

Installation instructions:

List any non-trivial installation instructions.

Testing instructions:

  1. Open page A
  2. Do thing B
  3. Expect C to happen
  4. If D happened instead - check failed.

Reviewers:

Merge checklist:

Post merge:

Author concerns:

List any concerns about this PR - inelegant solutions, hacks, quick-and-dirty implementations, concerns about migrations, etc.

HammadAhmadWaqas commented 3 years ago

@robrap I have incorporated feedback. Could you please review it again?