Open yarikoptic opened 7 months ago
FWIW, I have adjusted original gist from https://github.com/dandi/dandi-archive/issues/1891#issuecomment-2000305294 to include explicit page_size, it is https://gist.github.com/yarikoptic/314f9f419c7203bcc40331571cacf693 and here are timings for 100 (corresponds to default) and 1000:
100:
1000:
so seems to have the same runtime pretty much but throughput should be x10 more if we are getting 1000 assets at once I think.
Originally suggested as partial mediation to workaround
and (not really) discussed on slack. ATM we default to 100 assets, it leads to the need to page through 576 pages! If we increase to 1000 assets per request -- we reduce number of queries 10x fold. And that is already how e.g. https://neurosift.app does it -- sets page_size to 1000 (and stops on 10 pages ATM).
I think it would be really easy to test timing/performance to quantify benefit and "PR" should be quite easy to do .