How to obtain refresh token?

[aaronsteers]

I attempted to setup tap-spotify today but was unable to locate the needed refresh_token config value. From the Spotify Developer interface I see client_secret and client_id but not refresh_token.

Any help is much appreciated. Thanks!

[ReubenFrankel]

Hi AJ, thanks for opening this.

At the moment, there are a few hoops you need to jump through to get your refresh_token. I would suggest this approach from the Matatika documentation. Failing that, there is a nice blog post detailing some more manual steps, or you can refer to the Spotify Authorization Code Flow documentation directly ('Request User Authorization' and 'Request Access Token' sections).

There is an obvious need to update the README here, for those using this tap outside of the Matatika platform. This is definitely on our to-do list. 😄

Let me know if this helps - I'll leave this issue open for now, in case anyone else has a similar problem in the meantime.

[ReubenFrankel]

Anecdotally, I think this issue is a great example of how it can sometimes be difficult for a user to obtain the credentials needed to access their data - due to technical ability, time available or availability/accessibility of documentation (as in this case 😅).

One of our aims is to integrate OAuth sign-in across our supported taps from the Matatika app, which would eliminate the need for a user to manage any credentials designed for programatic access at all. You can see an example of how we have implemented proxy OAuth in our tap-googleads fork (oauth.-prefixed settings). The possibility of migrating some of this functionality to the Meltano SDK is something we are keen to discuss with the Meltano team, at some point in the future.

[aaronsteers]

@ReubenFrankel - Super helpful - thanks! I've started a PR against the README.md (still WIP) here: https://github.com/Matatika/tap-spotify/pull/5

[aaronsteers]

After generating a new token and providing it as refresh_token as described in my PR, I'm getting this error: RuntimeError: Failed OAuth login, response was '{'error': 'invalid_grant', 'error_description': 'Invalid refresh token'}'. 400 Client Error: Bad Request for url: https://accounts.spotify.com/api/token

Any ideas? Perhaps I'm missing one or more scopes?

j@ajs-macbook-pro aj-dataops-personal % meltano invoke tap-spotify --test       
2022-04-19T04:50:53.535802Z [info     ] Environment 'dev' is active
time=2022-04-18 21:50:54 name=tap-spotify level=INFO message=tap-spotify v0.3.0, Meltano SDK v0.3.18)
time=2022-04-18 21:50:54 name=tap-spotify level=INFO message=Skipping parse of env var settings...
time=2022-04-18 21:50:54 name=tap-spotify level=INFO message=Config validation passed with 0 errors and 0 warnings.
time=2022-04-18 21:50:54 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output.
time=2022-04-18 21:50:54 name=tap-spotify level=INFO message=Beginning full_table sync of 'global_top_tracks_daily_stream'...
time=2022-04-18 21:50:54 name=tap-spotify level=INFO message=Tap has custom mapper. Using 1 provided map(s).
{"type": "SCHEMA", "stream": "global_top_tracks_daily_stream", "schema": {"properties": {"album": {"properties": {"album_type": {"type": ["string", "null"]}, "artists": {"items": {"properties": {"external_urls": {"properties": {"spotify": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "followers": {"properties": {"href": {"type": ["string", "null"]}, "total": {"type": ["integer", "null"]}}, "type": ["object", "null"]}, "genres": {"items": {"type": ["string"]}, "type": ["array", "null"]}, "href": {"type": ["string", "null"]}, "id": {"type": ["string", "null"]}, "images": {"items": {"properties": {"height": {"type": ["integer", "null"]}, "url": {"type": ["string", "null"]}, "width": {"type": ["integer", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "name": {"type": ["string", "null"]}, "popularity": {"type": ["integer", "null"]}, "type": {"type": ["string", "null"]}, "uri": {"type": ["string", "null"]}, "rank": {"type": ["integer", "null"]}, "synced_at": {"format": "date-time", "type": ["string", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "available_markets": {"items": {"type": ["string"]}, "type": ["array", "null"]}, "external_urls": {"properties": {"spotify": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "href": {"type": ["string", "null"]}, "id": {"type": ["string", "null"]}, "images": {"items": {"properties": {"height": {"type": ["integer", "null"]}, "url": {"type": ["string", "null"]}, "width": {"type": ["integer", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "name": {"type": ["string", "null"]}, "release_date": {"type": ["string", "null"]}, "release_date_precision": {"type": ["string", "null"]}, "restrictions": {"properties": {"reason": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "total_tracks": {"type": ["integer", "null"]}, "type": {"type": ["string", "null"]}, "uri": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "artists": {"items": {"properties": {"external_urls": {"properties": {"spotify": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "followers": {"properties": {"href": {"type": ["string", "null"]}, "total": {"type": ["integer", "null"]}}, "type": ["object", "null"]}, "genres": {"items": {"type": ["string"]}, "type": ["array", "null"]}, "href": {"type": ["string", "null"]}, "id": {"type": ["string", "null"]}, "images": {"items": {"properties": {"height": {"type": ["integer", "null"]}, "url": {"type": ["string", "null"]}, "width": {"type": ["integer", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "name": {"type": ["string", "null"]}, "popularity": {"type": ["integer", "null"]}, "type": {"type": ["string", "null"]}, "uri": {"type": ["string", "null"]}, "rank": {"type": ["integer", "null"]}, "synced_at": {"format": "date-time", "type": ["string", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "available_markets": {"items": {"type": ["string"]}, "type": ["array", "null"]}, "disc_number": {"type": ["integer", "null"]}, "duration_ms": {"type": ["integer", "null"]}, "explicit": {"type": ["boolean", "null"]}, "external_ids": {"properties": {"ean": {"type": ["string", "null"]}, "isrc": {"type": ["string", "null"]}, "upc": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "external_urls": {"properties": {"spotify": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "href": {"type": ["string", "null"]}, "id": {"type": ["string", "null"]}, "is_local": {"type": ["boolean", "null"]}, "is_playable": {"type": ["boolean", "null"]}, "name": {"type": ["string", "null"]}, "popularity": {"type": ["integer", "null"]}, "preview_url": {"type": ["string", "null"]}, "restrictions": {"properties": {"reason": {"type": ["string", "null"]}}, "type": ["object", "null"]}, "track_number": {"type": ["integer", "null"]}, "type": {"type": ["string", "null"]}, "uri": {"type": ["string", "null"]}, "rank": {"type": ["integer", "null"]}, "synced_at": {"format": "date-time", "type": ["string", "null"]}}, "type": "object"}, "key_properties": ["rank", "synced_at"]}
Traceback (most recent call last):
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/authenticators.py", line 427, in update_access_token
    token_response.raise_for_status()
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://accounts.spotify.com/api/token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/bin/tap-spotify", line 8, in <module>
    sys.exit(TapSpotify.cli())
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 456, in cli
    tap.run_connection_test()
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 175, in run_connection_test
    stream.sync()
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1010, in sync
    self._sync_records(context)
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 937, in _sync_records
    for record_result in self.get_records(current_context):
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 396, in get_records
    for record in self.request_records(context):
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 290, in request_records
    prepared_request = self.prepare_request(
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 256, in prepare_request
    headers.update(authenticator.auth_headers or {})
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/authenticators.py", line 316, in auth_headers
    self.update_access_token()
  File "/Users/aj/Source/aj-dataops-personal/.meltano/extractors/tap-spotify/venv/lib/python3.8/site-packages/singer_sdk/authenticators.py", line 430, in update_access_token
    raise RuntimeError(
RuntimeError: Failed OAuth login, response was '{'error': 'invalid_grant', 'error_description': 'Invalid refresh token'}'. 400 Client Error: Bad Request for url: https://accounts.spotify.com/api/token

[ReubenFrankel]

I was under the impression that only the user-top-read scope was required, but @DanielPDWalker was able to get it working without any at all (could be a change on Spotify's end). The scopes you specified in your PR are not required.

We tried the Thunder Client VS Code extension and were only able to get the value of access_token as the generated token - not the required refresh_token value. Postman seems parse all fields from the response after clicking 'Get New Access Token'.

The request access token response should look like this:

{
    "access_token": ...
    "token_type": "Bearer",
    "expires_in": 3600,
    "refresh_token": ...
}

[ReubenFrankel]

I was under the impression that only the user-top-read scope was required, but @DanielPDWalker was able to get it working without any at all (could be a change on Spotify's end). The scopes you specified in your PR are not required.

Following up on this: user-top-read scope is required when obtaining a refresh token, in order to allow tap-spotify read access to a user's top items in these streams.

[ReubenFrankel]

Resolved by https://github.com/Matatika/tap-spotify/commit/d88c3c82b4ec422080670d1c91c1f8ab057d3e14 - see https://github.com/Matatika/tap-spotify/pull/5#issuecomment-1606114131.