search

Matatika / tap-veeqo

Singer tap for Veeqo
GNU Affero General Public License v3.0
2 stars 1 forks source link

feat: Better rate limit handling #76

Open ReubenFrankel opened 3 weeks ago

ReubenFrankel commented 3 weeks ago

There have been a number of times the tap has hit a 429 Too Many Requests error and the inbuilt back-off strategy has not been able to recover the sync. We should investigate if there is a better way to handle these situations, such as

  1. respecting a remaining rate limit value, if available (typically a header)
  2. simply increasing the max attempts (override backoff_max_tries)
Error logs ``` 2024-10-28T18:04:32.667482Z [info ] 2024-10-28 18:04:32,667 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.006067, "tags": {"stream": "sellables", "endpoint": "/sellables", "http_status_code": 429, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:32.667813Z [info ] 2024-10-28 18:04:32,667 | INFO | backoff | Backing off _request(...) for 2.2s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:32.668060Z [info ] 2024-10-28 18:04:32,667 | ERROR | root | Backing off 2.24 seconds after 1 tries calling function > with args (, None) and kwargs {} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:34.915149Z [info ] 2024-10-28 18:04:34,914 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.006491, "tags": {"stream": "sellables", "endpoint": "/sellables", "http_status_code": 429, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:34.915625Z [info ] 2024-10-28 18:04:34,914 | INFO | backoff | Backing off _request(...) for 4.4s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:34.915871Z [info ] 2024-10-28 18:04:34,914 | ERROR | root | Backing off 4.43 seconds after 2 tries calling function > with args (, None) and kwargs {} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:39.352907Z [info ] 2024-10-28 18:04:39,352 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.006668, "tags": {"stream": "sellables", "endpoint": "/sellables", "http_status_code": 429, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:39.354010Z [info ] 2024-10-28 18:04:39,352 | INFO | backoff | Backing off _request(...) for 8.5s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:39.354263Z [info ] 2024-10-28 18:04:39,352 | ERROR | root | Backing off 8.50 seconds after 3 tries calling function > with args (, None) and kwargs {} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:47.866144Z [info ] 2024-10-28 18:04:47,865 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.008413, "tags": {"stream": "sellables", "endpoint": "/sellables", "http_status_code": 429, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:47.866557Z [info ] 2024-10-28 18:04:47,865 | INFO | backoff | Backing off _request(...) for 16.7s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:04:47.866747Z [info ] 2024-10-28 18:04:47,865 | ERROR | root | Backing off 16.70 seconds after 4 tries calling function > with args (, None) and kwargs {} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.590515Z [info ] 2024-10-28 18:05:04,589 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.007571, "tags": {"stream": "sellables", "endpoint": "/sellables", "http_status_code": 429, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.591069Z [info ] 2024-10-28 18:05:04,590 | ERROR | backoff | Giving up _request(...) after 5 tries (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.591557Z [info ] 2024-10-28 18:05:04,590 | INFO | singer_sdk.metrics | METRIC: {"type": "counter", "metric": "http_request_count", "value": 61, "tags": {"stream": "sellables", "endpoint": "/sellables", "pid": 258}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.591870Z [info ] 2024-10-28 18:05:04,590 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "sync_duration", "value": 52.638946533203125, "tags": {"stream": "sellables", "pid": 258, "context": {}, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.592199Z [info ] 2024-10-28 18:05:04,590 | INFO | singer_sdk.metrics | METRIC: {"type": "counter", "metric": "record_count", "value": 671, "tags": {"stream": "sellables", "pid": 258, "context": {}}} cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.592414Z [info ] 2024-10-28 18:05:04,590 | ERROR | tap-veeqo.sellables | An unhandled error occurred while syncing 'sellables' cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.592638Z [info ] Traceback (most recent call last): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.592901Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1231, in sync cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.593101Z [info ] for _ in self._sync_records(context=context): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.593335Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1128, in _sync_records cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.593573Z [info ] for idx, record_result in enumerate(self.get_records(current_context)): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.593833Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 597, in get_records cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.594049Z [info ] for record in self.request_records(context): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.594276Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 409, in request_records cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.594486Z [info ] resp = decorated_request(prepared_request, context) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.594685Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.594937Z [info ] ret = target(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.595193Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 294, in _request cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.595415Z [info ] self.validate_response(response) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.595726Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 201, in validate_response cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.595945Z [info ] raise RetriableAPIError(msg, response) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.596202Z [info ] singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.596385Z [info ] Traceback (most recent call last): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.596584Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/bin/tap-veeqo", line 8, in cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.596905Z [info ] sys.exit(TapVeeqo.cli()) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.597108Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.597360Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.597549Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.597751Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.597989Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 82, in invoke cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.598151Z [info ] return super().invoke(ctx) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.598360Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.598561Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.598791Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.598974Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.599174Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 507, in invoke cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.599384Z [info ] tap.sync_all() cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.599586Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 466, in sync_all cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.599748Z [info ] stream.sync() cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.599918Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1231, in sync cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.600233Z [info ] for _ in self._sync_records(context=context): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.600504Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1128, in _sync_records cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.600711Z [info ] for idx, record_result in enumerate(self.get_records(current_context)): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.600991Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 597, in get_records cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.601231Z [info ] for record in self.request_records(context): cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.601499Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 409, in request_records cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.601677Z [info ] resp = decorated_request(prepared_request, context) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.601866Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.602038Z [info ] ret = target(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.602212Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 294, in _request cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.602404Z [info ] self.validate_response(response) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.602627Z [info ] File "/tmp/shelltaskscripts3219743586439785816/.meltano/extractors/tap-veeqo/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 201, in validate_response cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.602884Z [info ] raise RetriableAPIError(msg, response) cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo 2024-10-28T18:05:04.603132Z [info ] singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /sellables cmd_type=elb consumer=False job_name=dev:tap-veeqo-to-target-postgres:54b5b47e-efc1-4169-a278-8aae9a590cb6 name=tap-veeqo producer=True run_id=38201f4f-e882-448d-a7d6-cbece59e66a5 stdio=stderr string_id=tap-veeqo ```
ReubenFrankel commented 3 weeks ago
  1. simply increasing the max attempts (override backoff_max_tries)

5dd84d9de93eb46fac535db21acec395016f1207