search

biocore / redbiom

Sample search by metadata and features
Other
44 stars 20 forks source link

Support for large numbers of features #103

Closed wasade closed 2 years ago

wasade commented 3 years ago

This pull requests expands on the server side logic to support bulk load operations for obtaining indices from identifiers, and the load of identifier specific data.

The motivation is to support tables containing millions of identifiers. When performed individually, these operations require milliseconds per query, and 1 million times 1 millisecond begins to get large. What we're in effect doing here is packing in more data into the individual requests to reduce the HTTP request/response overhead.

wasade commented 2 years ago

Sorry, missed the batchsize comment. I don't think it's understood well enough to motivate centralizing. At this point, based on what I know, I suspect that effort will not be for gain.

wasade commented 2 years ago

Sorry, missed the batchsize comment. I don't think it's understood well enough to motivate centralizing. At this point, based on what I know, I suspect that effort will not be for gain.