search

riparias / gbif-alert

GBIF Alert is a GBIF occurrence based alert system.
https://gbif-alert-demo.thebinaryforest.net/
MIT License
8 stars 2 forks source link

Performance: investigate easy wins for the observation import? #243

Closed niconoe closed 1 year ago

niconoe commented 1 year ago

The import script has never been thoroughly inspected for performance, because it was good enough for now.

Now, with 500.000+ observations in the LIFE RIPARIAS instance (which uses a cheap RDS database plan), the full import takes more than 3 hours. We didn't received complains so far because it's running nightly, but it might be good if there are some "easy" optimizations that could already improve this.

niconoe commented 1 year ago

Update: I made some good improvements on the import_per_improvements branch (import time halved on my laptop).

The changes were more invasive than what I hoped, so it should be carefully tested before deployment.

niconoe commented 1 year ago

Update: since that became more urgent due to #259, I implemented all those changes, import performances seems satisfactory again.