paleobiodb / data_service

The PBDB Data Service, API and table/system maintenance scripts
Artistic License 2.0
12 stars 0 forks source link

Pre-compute full diversity method for whole db #18

Open vjpsyverson opened 7 years ago

vjpsyverson commented 7 years ago

Now that the full diversity method is exposed in Navigator (probably going to go to production soon as no-one has yet found any bugs in it), it would be helpful to have a pre-computed result available to the API for that method on the whole database so that it doesn't have to recompute every time someone clicks through to that window without filtering.

markuhen commented 7 years ago

Perhaps in the meantime, you could add a warning when someone tries to do the whole database and ask them if they really want to do that!

aazaff commented 7 years ago

I wrote a shiny app a while back for calculating standardized richness time-series using the API. I found that the longest part was downloading the data, even for relatively smaller subsets of the data - e.g., Cenozoic mollusks. Perhaps pre-calculating richness for a number of common intervals and taxonomic groups would make sense.

vjpsyverson commented 7 years ago

If we want to do that for some of the most common intervals and taxonomic groups, I suggest that we start with the whole database and then start logging all the requests so that we can figure out what taxa/intervals are most commonly requested and maybe consider pre-computing those.