It was previously thought that directly integrating the async throttle with Porter was not needed because we can just throttle high level Porter import operations. However, this is false, for two reasons:
Internally, fetches can be retried (by default up to 5 times).
Any given import may pull down any number of resources to satisfy the import operation. The most common case is enumerating a paginated resource that results in n requests for n pages.
Each of these additional requests must be throttled independently to avoid triggering limits, whether a retry or the next resource in a sequence. For this to be possible, the throttle must be integrated into ImportConnector so it can throttle transparently without burdening the developer with additional calls or configuration.
A default throttle should be provided for async imports but it should be possible to override with a custom configuration or implementation via AsyncImportSpecification. Throttling will not be available for sync imports until such a time as the sync API converges with the async API internally.
It was previously thought that directly integrating the async throttle with Porter was not needed because we can just throttle high level Porter import operations. However, this is false, for two reasons:
Each of these additional requests must be throttled independently to avoid triggering limits, whether a retry or the next resource in a sequence. For this to be possible, the throttle must be integrated into
ImportConnectorso it can throttle transparently without burdening the developer with additional calls or configuration.A default throttle should be provided for async imports but it should be possible to override with a custom configuration or implementation via
AsyncImportSpecification. Throttling will not be available for sync imports until such a time as the sync API converges with the async API internally.