search

turbot / steampipe-plugin-sdk

Steampipe Plugin SDK is a simple abstraction layer to write a Steampipe plugin. Plugins automatically work across all engine types including the Steampipe CLI, Postgres FDW, SQLite extension and the export CLI.
https://hub.steampipe.io/plugins
Apache License 2.0
30 stars 14 forks source link

could plugins tell the engine about rate limits? #194

Open judell opened 2 years ago

judell commented 2 years ago

Having implemented (experimental) workarounds for rate-limiting in three different plugins, I'm coming to the view that it would be great if the plugin writer could tell the engine what's known about an API's rate-limiting policy, and the plugin sdk could deal with it. Here are the three cases:

  1. GitHub (table_github_commit, table_github_issue): I implemented retryHydrate, and it does overcome the limit, but suboptimally. When GH limits you, it tells you exactly when you can resume your next hourly batch of 5K calls. As plugin writer I'd just want to tell the engine about the rule: when throttled, get the reset GH tells you and wait until then.

  2. Slack (table_slack_search): I implemented retryHydrate and encoded the rule to wait 3 secs, as Slack tells you to do when throttled. As plugin writer I'd just like to tell the engine about the rule.

  3. GoogleWorkspace (table_gmail_message): What worked here for me, to cache 138K message in a matview, was to reduce MaxConcurrency on Get from 50 to 5. I wouldn't always want things to slow down that way, only when I'm asking for more data than the per-minute limit allows. As plugin writer I'd like to tell the engine: the per-minute limit is roughly x, go as fast as you can, then adjust when throttled.

There are surely more patterns than these, so it may not yet be possible to sketch how the plugin writer would write the rate-limiting policy that the plugin sdk would try to implement. But it's a nice aspiration.

Related: https://github.com/turbot/steampipe-plugin-github/issues/118, https://github.com/turbot/steampipe-plugin-slack/issues/23, https://github.com/turbot/steampipe-postgres-fdw/issues/133

christianherweg0807 commented 2 years ago

Would be great to handle API limits gracefully.

massyn commented 1 year ago

Seems this is a more common issue - this appears to be a duplicate of the enhancement request I raised - https://github.com/turbot/steampipe-plugin-sdk/issues/595