immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
52.62k stars 2.79k forks source link

feat(server): faster geodata import #14241

Closed mertalev closed 5 days ago

mertalev commented 5 days ago

Description

This PR makes a number of optimizations to make the geodata import faster.

Also changes the serial column in the natural earth table to a generated identity column. This is both needed for the atomic table rename/drop to work and follows official recommendations to avoid serial.

Testing

These changes made the import go from 18s on an SSD to 8s 6.5s. The impact of this change will be especially noticeable on hard drives and other slow media.

mertalev commented 5 days ago

The only thing I am not sure about is creating the indexes afterwards. In my testing previously that was a good bit slower in a resource constrained environment.

It's faster to do it at the end by about 800ms from my testing. It's also better in that it lets us set fillfactor to 100 - it doesn't play nicely with inserts.

I'd be interested to learn why we need a public version of the function for it to be indexable...

I kept getting the type "earth" does not exist error when I tried to use ll_to_earth for the expression index. I'm guessing it's something to do with the search path being different during index creation than in normal queries. A neat benefit of using the explicit version is that we might not need the sed hack for backups anymore.