search

MeltanoLabs / tap-github

A Singer tap for extracting data from Github. Powered by the Meltano SDK for Singer Taps: https://sdk.meltano.com
Apache License 2.0
16 stars 27 forks source link

Hard to tell if API token is valid or not #222

Open visch opened 1 year ago

visch commented 1 year ago

https://stackoverflow.com/questions/22438805/github-api-oauth-token-validation offers an option to check on the x-ratelimit-limit header to see if your API token is valid or not. Could be an interesting way to tell folks if their token is valid as right now it's not clear if the token doesn't have access to a specific endpoint or if the api token itself is just invalid.

From this slack thread https://meltano.slack.com/archives/C01TCRBBJD7/p1689349341108699

visch commented 3 months ago

Another example hit today.

To repeat

  1. Provide an invalid token
  2. You get an error like this
2024-03-28 09:33:10,071 | WARNING  | tap-github           | A token was dismissed. 401 Client Error: b'{"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"}' (Reason: Unauthorized)
2024-03-28 09:33:10,071 | INFO     | tap-github           | Tap will run with 0 auth tokens
2024-03-28 09:33:10,072 | INFO     | tap-github           | No auth token detected. For higher rate limits, please specify `auth_token` in config.
2024-03-28 09:33:10,225 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.153058, "tags": {"stream": "tempStream", "endpoint": "", "http_status_code": 403, "status": "failed"}}
2024-03-28 09:33:10,225 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 0, "tags": {"stream": "tempStream", "endpoint": "", "context": {}}}
2024-03-28 09:33:10,225 | ERROR    | tap-github           | An unhandled error occurred while syncing 'repositories'
Traceback (most recent call last):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync
    for _ in self._sync_records(context=context):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1054, in _sync_records
    context_list = [context] if context is not None else self.partitions
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 175, in partitions
    augmented_repo_list += self.get_repo_ids(
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 119, in get_repo_ids
    for record in temp_stream.request_records({}):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 397, in request_records
    resp = decorated_request(prepared_request, context)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 276, in _request
    self.validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 103, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 440, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 213, in validate_response
    self.authenticator.get_next_auth_token()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/authenticator.py", line 214, in get_next_auth_token
    raise RuntimeError(
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.
Traceback (most recent call last):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/bin/tap-github", line 8, in <module>
    sys.exit(cli())
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 500, in invoke
    tap.sync_all()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 459, in sync_all
    stream.sync()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1186, in sync
    raise ex
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync
    for _ in self._sync_records(context=context):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1054, in _sync_records
    context_list = [context] if context is not None else self.partitions
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 175, in partitions
    augmented_repo_list += self.get_repo_ids(
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 119, in get_repo_ids
    for record in temp_stream.request_records({}):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 397, in request_records
    resp = decorated_request(prepared_request, context)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 276, in _request
    self.validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 103, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 440, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 213, in validate_response
    self.authenticator.get_next_auth_token()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/authenticator.py", line 214, in get_next_auth_token
    raise RuntimeError(
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.

This runtime error is highly misleading. The github token that got a 401 should most likely fail the tap (Unless there's some reason a 401 is given for something that's needed to retry). Maybe this is related to https://github.com/MeltanoLabs/tap-github/issues/243 ?