It turns out (after very helpful conversations with @danielballan, thanks!) that one can dramatically speed up iterative fetching of Databroker 2 (Tiled backed) documents by bulk-downloading, for instance:
This PR takes this MCVE and implements it in SST1RSoXSDB.summarize_run, a worker function for doing Tiled searches.
Other changes:
tidies up a lot of the code within this function to improve readability and remove some unnecessary things.
takes several warnings that are entirely related to input validation errors within the user's control and promotes them to raise a real ValueError.
adds a flag (default on) that buries all "field could not be found" errors behind a single warning. Otherwise every time metadata changes the output of the function fills a terminal with hundreds of warnings...
adds a special case so the run start time is returned as a datetime.datetime rather than an epoch number.
Resulting performance: a 1184-entry search which used to take 1m 5s now completes in about 14s, filling ~80 runs / second. In testing tonight (when the Tiled server is under less load) I am seeing performance up to ~120 runs / second.
It turns out (after very helpful conversations with @danielballan, thanks!) that one can dramatically speed up iterative fetching of Databroker 2 (Tiled backed) documents by bulk-downloading, for instance:
SST1RSoXSDB.summarize_run, a worker function for doing Tiled searches.Other changes:
datetime.datetimerather than an epoch number.Resulting performance: a 1184-entry search which used to take 1m 5s now completes in about 14s, filling ~80 runs / second. In testing tonight (when the Tiled server is under less load) I am seeing performance up to ~120 runs / second.