anelendata / tap-rest-api

Singer.io tap for generic Rest API
Apache License 2.0
23 stars 14 forks source link

Wrong params for singer.utils.backoff decorator #27

Closed daigotanaka closed 2 years ago

daigotanaka commented 2 years ago

Python backoff module lets the code retry in a specified interval for given maximum retries when it encounters an exception such as HTTP server errors (5xx).

tap-rest-api uses singer wrapped version references as singer.utils.backoff.

But the parameters are set as though it is using the native backoff

@utils.backoff((backoff.expo, requests.exceptions.RequestException), _giveup)
@utils.ratelimit(20, 1)
def generate_request(stream_id, url, auth_method="no_auth", headers=None,
                     username=None, password=None):

https://github.com/anelendata/tap-rest-api/blob/master/tap_rest_api/helper.py#L301

This results in TypeError: catching classes that do not inherit from BaseException is not allowed error in the backoff routine:

CRITICAL catching classes that do not inherit from BaseException is not allowed
--
Traceback (most recent call last):
File "/app/workspace/proc_01/lib/python3.6/site-packages/backoff/_sync.py", line 94, in retry
ret = target(*args, **kwargs)
File "/app/workspace/proc_01/lib/python3.6/site-packages/singer/utils.py", line 95, in wrapper
return func(*args, **kwargs)
File "/app/workspace/proc_01/lib/python3.6/site-packages/tap_rest_api/helper.py", line 321, in generate_request
resp.raise_for_status()
File "/app/workspace/proc_01/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://xxxx
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/workspace/proc_01/bin/tap-rest-api", line 8, in <module>
sys.exit(main())
File "/app/workspace/proc_01/lib/python3.6/site-packages/singer/utils.py", line 229, in wrapped
return fnc(*args, **kwargs)
File "/app/workspace/proc_01/lib/python3.6/site-packages/tap_rest_api/__init__.py", line 191, in main
auth_method, raw=args.raw, filter_by_schema=filter_by_schema)
File "/app/workspace/proc_01/lib/python3.6/site-packages/tap_rest_api/sync.py", line 211, in sync
raise e
File "/app/workspace/proc_01/lib/python3.6/site-packages/tap_rest_api/sync.py", line 208, in sync
filter_by_schema=filter_by_schema)
File "/app/workspace/proc_01/lib/python3.6/site-packages/tap_rest_api/sync.py", line 98, in sync_rows
config.get("password"))
File "/app/workspace/proc_01/lib/python3.6/site-packages/backoff/_sync.py", line 95, in retry
except exception as e:
TypeError: catching classes that do not inherit from BaseException is not allowed