Django offers a nice loaddata management command, but one downside is that it does not offer bulk creation of objects, so it can be quite slow.
We've implemented a few data pipelines now that do some light extraction and transformation, then use the bulk_create method to insert the fixture data into the database.
This is a nice pattern that we might consider making reusable. I'm curious, too, whether there are any Django plugins that facilitate bulk model instance creation.
Proposal
Learn more about why loaddata does not allow for bulk creation. @jeancochrane, you mentioned there was a Django ticket on this subject, but I couldn't find it during a cursory search. Do you have it handy, by chance?
Research any existing plug-ins for bulk loading model instances from a fixture.
Depending on the outcome of points one and two, the third step could be submitting a Django PR, piloting and adopting a plugin that offers the behavior we want, or implementing our own plugin based on the bulk loading management commands we've already written.
Deliverables
This R&D effort will yield increased understanding of data loading considerations, as well as reusable code to facilitate bulk loading of data into our applications (though, that could take one of several forms, listed in the third point, above).
Timeline
I'd expect research to take about half a day, and then adopting an existing tool or writing fresh code to take between one and three days, depending on which avenue we take.
FWIW, I'd also say this is low priority, just wanted to capture a conversation I had with @jeancochrane before they leave us.
Background
Django offers a nice
loaddatamanagement command, but one downside is that it does not offer bulk creation of objects, so it can be quite slow.We've implemented a few data pipelines now that do some light extraction and transformation, then use the
bulk_createmethod to insert the fixture data into the database.This is a nice pattern that we might consider making reusable. I'm curious, too, whether there are any Django plugins that facilitate bulk model instance creation.
Proposal
loaddatadoes not allow for bulk creation. @jeancochrane, you mentioned there was a Django ticket on this subject, but I couldn't find it during a cursory search. Do you have it handy, by chance?Deliverables
This R&D effort will yield increased understanding of data loading considerations, as well as reusable code to facilitate bulk loading of data into our applications (though, that could take one of several forms, listed in the third point, above).
Timeline
I'd expect research to take about half a day, and then adopting an existing tool or writing fresh code to take between one and three days, depending on which avenue we take.
FWIW, I'd also say this is low priority, just wanted to capture a conversation I had with @jeancochrane before they leave us.