search

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.
https://vast-survey.org/vast-pipeline/
MIT License
7 stars 3 forks source link

Count queries are expensive #605

Closed marxide closed 1 year ago

marxide commented 2 years ago

The web interface executes several queries to display some counts. On the index page, these include the number of runs, images, measurements, and sources. Similar counts are run on other pages too, such as the run detail page.

Given the size of our database, these queries can be expensive to run. The index page count queries can take longer than a minute to complete before serving the page to the user.

These counts are not important – they're just nice to have. We should remove or optimize them. One easy optimization would be to calculate the counts at the end of a run and store them in the Run model. Then, to display the counts, we only need to query for the stored counts rather than count the relevant rows in the database tables.

marxide commented 2 years ago

Some of the counts are already stored in the Run model, but summing the measurement counts for all Run objects will count many measurements more than once as measurements can belong to more than one run.