Open quasistoic opened 1 year ago
This is as fast as we can reasonably get the initial load without getting rid of the score-based sorting.
Reopening. Turns out I'm not actually being rate-limited yet, but the calls themselves are I/O-bound, which means I should be able to parallelize. Additionally passing the --vault argument for each get call reportedly would improve latency.
With some experimentation, I was able to infer that the 300th item get call in a single minute trips me over whatever actual qpm limits the 1Password API uses, so this allows me to reduce what used to take about 10-15 minutes down to about 3 minutes (with N~800 items).
So, significant speedup, but still not quite fast enough to ditch the on-disk cache.
Relevant commits:
I was hoping that 5cc958cf30d3b6470d91bd47857224a155bdea1c would result in fewer backend API calls made by the op command line when running an item get
, so I attempted to raise my throttled fetch rate to around 400qpm, but after exactly 300 successful fetches, I was blocked for an hour, so I suppose we're stuck with under 300qpm.
Initial data download takes a very long time because it has to make a ton of API calls, one per item in 1Password. I plan to switch the first call to “item list” to provide json output, which includes enough details that I can get rid of all the per-item calls when composing the list view, so the initial loading will change from many minutes to a few seconds. This should obviate the need for me to do on-disk caching. Also just makes the whole experience more pleasant.