perf(backend): remove null values before storing processed data in database - Githubissues

loculus-project / loculus

An open-source software package to power microbial genomic databases

https://loculus.org

GNU Affero General Public License v3.0

37 stars 2 forks source link

perf(backend): remove null values before storing processed data in database #3233

Closed corneliusroemer closed 3 days ago

corneliusroemer commented 3 days ago

resolves #2793

preview URL: https://dont-save-nulls.loculus.org

Summary

Don't store fields with null values in processed data to improve performance (reduce storage, size of data to transfer into and out of db etc.)
This is possible now that we impute missing fields as null when preparing get-released-data, see #3232

Screenshot

No longer any nulls in processed data:

pgAdmin 4 2024-11-18 23 41 58

PR Checklist

[ ] All necessary documentation has been adapted.
[ ] The implemented feature is covered by an appropriate test.