search

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.
https://vast-survey.org/vast-pipeline/
MIT License
8 stars 3 forks source link

Improve the bulk_create using django-postgres-copy #383

Open srggrs opened 3 years ago

srggrs commented 3 years ago

bulk_create can be improved by using django-postgres-copy as suggested in https://pmbaumgartner.github.io/blog/the-fastest-way-to-load-data-django-postgresql/ also would be great writing some test using Faker as showed in the article.

ajstewart commented 3 years ago

I've realised the problem with this method is that you cannot return the primary key values when doing the COPY command.

I'm not sure what the best way would be in order to obtain the id values if we were to use this method - in that how would you guarantee you had the right id for the right object.