etianen
commented
3 years ago Thanks for this, and for the tests.
I'm really surprised this works! Performing a count on a large dataset can be very slow, and chunking through a dataset gets slower each chunk due to how databases perform offsetting.
But, pragmatically, for databases that don't support server side cursors, it's better than nothing.
I would suggest the following changes:
- Lose the count. Just keep slicing until you get no results.
- Rather than using offset, order the queryset by pk (asc), and after each batch, filter the next batch by pk__gt=final_pk_of_prev_batch
(2) is really important. It means this will scale to larger datasets, and it means, for auto-increment pks, no models will be missed if the data is being edited.
iJohnMaged
danihodovic
Came across your library and wanted to integrate it for a client, great work!
However when I was deploying on a relatively big database (2M rows, big model with lots of text data), the process was always getting killed on PythonAnywhere while using all the CPU and ram available, without creating a single index in
watson_searchentry.So I tinkered a bit and found that .iterator() is the issue in my case (limited resources, MySQL database too),
buildwatsondoesn't get to create any index, eventually changed the code to slice instead of .iterator and it got through.I add an argument to
buildwatsoncalled--slice-querysetto slice it instead of iterate, if that works for others in some cases.