dandi / dandi-archive

DANDI API server and Web app
https://dandiarchive.org
13 stars 11 forks source link

Boost default page size for `/assets/` endpoint #1907

Open yarikoptic opened 7 months ago

yarikoptic commented 7 months ago

Originally suggested as partial mediation to workaround

and (not really) discussed on slack. ATM we default to 100 assets, it leads to the need to page through 576 pages! If we increase to 1000 assets per request -- we reduce number of queries 10x fold. And that is already how e.g. https://neurosift.app does it -- sets page_size to 1000 (and stops on 10 pages ATM).

I think it would be really easy to test timing/performance to quantify benefit and "PR" should be quite easy to do .

yarikoptic commented 7 months ago

FWIW, I have adjusted original gist from https://github.com/dandi/dandi-archive/issues/1891#issuecomment-2000305294 to include explicit page_size, it is https://gist.github.com/yarikoptic/314f9f419c7203bcc40331571cacf693 and here are timings for 100 (corresponds to default) and 1000:

100:

response_times-page_size=100

1000:

response_times-page_size=1000

so seems to have the same runtime pretty much but throughput should be x10 more if we are getting 1000 assets at once I think.