Materials-Consortia / optimade-python-tools

Tools for implementing and consuming OPTIMADE APIs in Python
https://www.optimade.org/optimade-python-tools/
MIT License
64 stars 41 forks source link

Potential optimisations for requests that do not ask for a filter #2093

Open ml-evs opened 1 month ago

ml-evs commented 1 month ago

e.g., on server startup, take a series of chunked blobs corresponding to the default sort order of the API, then if a user requests with no filter and no page limit, return this static chunked data directly.

cc @JonathanSchmidt1 @eimrek (maybe this is a useful optimisation for e.g., the Li-ion set too)

eimrek commented 1 month ago

thanks @ml-evs, with basic indexes in mongodb, the default load can be made fairly fast except for the total number of structures (data_returned). Might be worth adding this just for that number. Or alternatively use the mongodb estimated document count for the filterless query (but i guess this can be wrong is some very rare cases?)

ml-evs commented 1 month ago

Maybe this is a good idea after all, if a web client always hits the same request, we could cache these as JSON files that are served statically rather than via the API, e.g., via the config file:

"static_index": ["?page_limit=100&response_fields=id&page_offset=0"]