quasistoic / op-tools

Tools for 1Password
1 stars 0 forks source link

Initial data load takes a long time #4

Open quasistoic opened 1 year ago

quasistoic commented 1 year ago

Initial data download takes a very long time because it has to make a ton of API calls, one per item in 1Password. I plan to switch the first call to “item list” to provide json output, which includes enough details that I can get rid of all the per-item calls when composing the list view, so the initial loading will change from many minutes to a few seconds. This should obviate the need for me to do on-disk caching. Also just makes the whole experience more pleasant.

quasistoic commented 1 year ago

This is as fast as we can reasonably get the initial load without getting rid of the score-based sorting.

quasistoic commented 1 year ago

Reopening. Turns out I'm not actually being rate-limited yet, but the calls themselves are I/O-bound, which means I should be able to parallelize. Additionally passing the --vault argument for each get call reportedly would improve latency.

quasistoic commented 1 year ago

With some experimentation, I was able to infer that the 300th item get call in a single minute trips me over whatever actual qpm limits the 1Password API uses, so this allows me to reduce what used to take about 10-15 minutes down to about 3 minutes (with N~800 items).

So, significant speedup, but still not quite fast enough to ditch the on-disk cache.

Relevant commits:

  1. 04e5b32061cc7f806e768880dc85fc0533f95c07
  2. 30413d835e4d8d343af9c04944e77e6a0679ae2c
  3. a8ca0fdb7991a1d34743a5f433ded81aa803ce5f
quasistoic commented 1 year ago

I was hoping that 5cc958cf30d3b6470d91bd47857224a155bdea1c would result in fewer backend API calls made by the op command line when running an item get, so I attempted to raise my throttled fetch rate to around 400qpm, but after exactly 300 successful fetches, I was blocked for an hour, so I suppose we're stuck with under 300qpm.