MeltanoLabs / tap-gitlab

Singer.io Tap for extracting data from Gitlab's API
GNU Affero General Public License v3.0
11 stars 29 forks source link

feat: add on_predicate decorator to handle 429 #102

Closed AlejandroUPC closed 5 months ago

AlejandroUPC commented 5 months ago

Description

This MR aims to cover the case HTTP 429 "Too Many Requests" (RFC685)i n a more specific way rather than the current way to handle any exception between the HTTP status code 400 and 500. This is motivated by the issue opened by @kgpayne here.

Implementation

The first decorator to be executed and added on top will be only evaluated when the HTTP code is 429 and then backoff whatever time is in the Retry-After header that gitlab returns (see here). The example is literally an implementation of the example supplied in the README.md from the backoff library here + adding the same amount of max_times.

sonarcloud[bot] commented 5 months ago

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

edgarrmondragon commented 5 months ago

Thanks @AlejandroUPC!