mysociety / mapit

A web service to map postcodes to administrative boundaries and more
Other
269 stars 88 forks source link

Faster ONSPD import #407

Closed dracos closed 1 year ago

dracos commented 1 year ago

This switches from the default loop/check/save to postgres copy into a temporary table and update/insert based on differences. The first run compared to previous way will have a lot of minute updates due to the location calculation being slightly different (going from CSV E/N -> 4326 in DB rather than code), but in the order of millimetres(!). Import time goes from 12 hours to a few minutes.