search

mountetna / magma

Data server with friendly data loaders
GNU General Public License v2.0
5 stars 2 forks source link

Table pagination is very slow #64

Open graft opened 6 years ago

graft commented 6 years ago

The Magma /retrieve endpoint can stream data in TSV format; in order to do this Magma computes page bounds with one SQL query, then queries for records within those bounds with another SQL query. The latter query is always fast; the former, however, becomes extremely slow when retrieving records from a large table.

graft commented 6 years ago

@dabrau this is perhaps solveable with partitioning?

dabrau commented 6 years ago

@graft partitioning will help if the query needs to access a subset of the child tables. So I think it will probably only make a difference in certain queries depending on how the table is partitioned.