search

dbt-labs / tap-ringcentral

Singer.io tap for RingCentral
Apache License 2.0
5 stars 13 forks source link

OAuth token expires if tap runs longer than 1 hour. #6

Open wss-chadical opened 4 years ago

wss-chadical commented 4 years ago
  1. Using tap via Stitch.
  2. Set our initial historical load data to 1-1-2019
  3. Failed multiple times, and looking at the logs it always happened exactly one hour into the extraction.
  4. RC OAuth access tokens expire after 1 hour. https://forums.developers.ringcentral.com/articles/678/how-to-avoid-expiration-of-access-token.html

Start of extraction

2020-01-21 22:50:24,665Z   main - INFO Running tap-ringcentral version 0.0.2 and target-stitch version 3.0.1
2020-01-21 22:50:24,750Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-ringcentral --config /tmp/tap_discover_config.json --discover
2020-01-21 22:50:25,352Z    tap - INFO Starting discovery.
2020-01-21 22:50:25,386Z   main - INFO Tap exited normally.
2020-01-21 22:50:25,386Z   main - INFO Saving list of discovered streams
2020-01-21 22:50:25,437Z   main - INFO Saving structure of stream contacts
2020-01-21 22:50:25,453Z   main - INFO Saving structure of stream call_log
2020-01-21 22:50:25,506Z   main - INFO Saving structure of stream company_call_log
2020-01-21 22:50:25,527Z   main - INFO Saving structure of stream messages
2020-01-21 22:50:25,545Z   main - INFO Writing catalog to file
2020-01-21 22:50:25,553Z   main - INFO Selected fields for stream company_call_log: duration, from, startTime, type, to, direction, sessionId, _contact_id, message, deleted, reason, extension, reasonDescription, action, id, result
2020-01-21 22:50:25,553Z   main - INFO Selected fields for stream contacts: account, firstName, name, phoneNumbers, email, type, jobTitle, lastName, status, extensionNumber, id
2020-01-21 22:50:25,553Z   main - INFO Selected fields for stream call_log: duration, from, startTime, type, to, direction, sessionId, _contact_id, message, deleted, reason, extension, reasonDescription, action, id, result
2020-01-21 22:50:25,595Z   main - INFO Current state: {"bookmarks": {"call_log": {"last_record": "2019-02-05T00:00:00Z", "field": "last_record"}}}
2020-01-21 22:50:25,595Z   main - INFO Starting tap: tap-env/bin/tap-ringcentral --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2020-01-21 22:50:25,598Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2020-01-21 22:50:26,196Z target - INFO Using batch_size_prefernces of {'user_batch_size_preference': None, 'batch_size_preference': None, 'full_table_streams': []}
2020-01-21 22:50:26,198Z target - INFO Using turbo_boost_factor of 1
2020-01-21 22:50:26,223Z    tap - INFO Starting sync.
2020-01-21 22:50:26,223Z    tap - INFO 'messages' is not marked selected, skipping.
2020-01-21 22:50:26,223Z    tap - INFO Syncing stream contacts with ContactsStream
2020-01-21 22:50:26,223Z    tap - INFO Syncing data for entity contacts (page=1)
2020-01-21 22:50:26,223Z    tap - INFO Making GET request to https://platform.ringcentral.com/restapi/v1.0/account/~/directory/entries ({'per_page': 1000, 'page': 1})
2020-01-21 22:50:26,516Z    tap - INFO Got status code 200

... 1 hour later ...

2020-01-21 23:50:30,651Z    tap - INFO Got status code 401
2020-01-21 23:50:31,542Z    tap - INFO Making GET request to https://platform.ringcentral.com/restapi/v1.0/account/~/extension/2299533011/call-log ({'showDeleted': True, 'dateTo': '2019-03-19T00:00:00+00:00', 'dateFrom': '2019-03-12T00:00:00+00:00', 'page': 1, 'perPage': 100})
2020-01-21 23:50:31,815Z    tap - INFO Got status code 401
2020-01-21 23:50:37,023Z    tap - INFO Making GET request to https://platform.ringcentral.com/restapi/v1.0/account/~/extension/2299533011/call-log ({'showDeleted': True, 'dateTo': '2019-03-19T00:00:00+00:00', 'dateFrom': '2019-03-12T00:00:00+00:00', 'page': 1, 'perPage': 100})
2020-01-21 23:50:37,298Z    tap - INFO Got status code 401
2020-01-21 23:50:45,563Z    tap - INFO Making GET request to https://platform.ringcentral.com/restapi/v1.0/account/~/extension/2299533011/call-log ({'showDeleted': True, 'dateTo': '2019-03-19T00:00:00+00:00', 'dateFrom': '2019-03-12T00:00:00+00:00', 'page': 1, 'perPage': 100})
2020-01-21 23:50:45,844Z    tap - INFO Got status code 401
2020-01-21 23:51:11,306Z    tap - INFO Making GET request to https://platform.ringcentral.com/restapi/v1.0/account/~/extension/2299533011/call-log ({'showDeleted': True, 'dateTo': '2019-03-19T00:00:00+00:00', 'dateFrom': '2019-03-12T00:00:00+00:00', 'page': 1, 'perPage': 100})
2020-01-21 23:51:11,578Z    tap - INFO Got status code 401
2020-01-21 23:51:11,579Z    tap - ERROR {
2020-01-21 23:51:11,579Z    tap -   "errorCode" : "TokenInvalid",
2020-01-21 23:51:11,579Z    tap -   "message" : "Token not found",
2020-01-21 23:51:11,579Z    tap -   "errors" : [ {
2020-01-21 23:51:11,579Z    tap -     "errorCode" : "OAU-213",
2020-01-21 23:51:11,579Z    tap -     "message" : "Token not found"
2020-01-21 23:51:11,579Z    tap -   } ]
2020-01-21 23:51:11,579Z    tap - }
2020-01-21 23:51:11,579Z    tap - ERROR Failed to sync endpoint call_log, moving on!
2020-01-21 23:51:11,579Z    tap - CRITICAL {
2020-01-21 23:51:11,580Z    tap -   "errorCode" : "TokenInvalid",
2020-01-21 23:51:11,580Z    tap -   "message" : "Token not found",
2020-01-21 23:51:11,580Z    tap -   "errors" : [ {
2020-01-21 23:51:11,580Z    tap -     "errorCode" : "OAU-213",
2020-01-21 23:51:11,580Z    tap -     "message" : "Token not found"
2020-01-21 23:51:11,580Z    tap -   } ]
2020-01-21 23:51:11,580Z    tap - }
2020-01-21 23:51:11,580Z    tap - Traceback (most recent call last):
2020-01-21 23:51:11,580Z    tap -   File "tap-env/bin/tap-ringcentral", line 9, in <module>
2020-01-21 23:51:11,580Z    tap -     load_entry_point('tap-ringcentral==0.0.2', 'console_scripts', 'tap-ringcentral')()
2020-01-21 23:51:11,580Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 192, in wrapped
2020-01-21 23:51:11,580Z    tap -     return fnc(*args, **kwargs)
2020-01-21 23:51:11,580Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/__init__.py", line 66, in main
2020-01-21 23:51:11,580Z    tap -     runner.do_sync()
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/__init__.py", line 42, in do_sync
2020-01-21 23:51:11,581Z    tap -     raise e
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/__init__.py", line 32, in do_sync
2020-01-21 23:51:11,581Z    tap -     stream.sync()
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_framework/streams.py", line 138, in sync
2020-01-21 23:51:11,581Z    tap -     return self.sync_data()
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/streams/base.py", line 82, in sync_data
2020-01-21 23:51:11,581Z    tap -     self.sync_data_for_period(date, interval)
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/streams/base.py", line 90, in sync_data_for_period
2020-01-21 23:51:11,581Z    tap -     self.sync_data_for_extension(date, interval, extensionId)
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/streams/base.py", line 142, in sync_data_for_extension
2020-01-21 23:51:11,581Z    tap -     url, self.API_METHOD, params=params, body=body)
2020-01-21 23:51:11,581Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/backoff.py", line 286, in retry
2020-01-21 23:51:11,581Z    tap -     ret = target(*args, **kwargs)
2020-01-21 23:51:11,582Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_ringcentral/client.py", line 79, in make_request
2020-01-21 23:51:11,582Z    tap -     raise APIException(response.text)
2020-01-21 23:51:11,582Z    tap - tap_ringcentral.client.APIException: {
2020-01-21 23:51:11,582Z    tap -   "errorCode" : "TokenInvalid",
2020-01-21 23:51:11,582Z    tap -   "message" : "Token not found",
2020-01-21 23:51:11,582Z    tap -   "errors" : [ {
2020-01-21 23:51:11,582Z    tap -     "errorCode" : "OAU-213",
2020-01-21 23:51:11,582Z    tap -     "message" : "Token not found"
2020-01-21 23:51:11,582Z    tap -   } ]
2020-01-21 23:51:11,582Z    tap - }
drewbanin commented 4 years ago

Thanks for the report @wss-chadical - sounds like we'll want to catch this request error and re-auth with ringcentral. I think that should be as easy as calling this method in the exception handler:

https://github.com/fishtown-analytics/tap-ringcentral/blob/0b2737cf6b2b2ea855e68178849669687de785ba/tap_ringcentral/client.py#L24

Is this something you'd be interested in contributing a fix for?

wss-chadical commented 4 years ago

@drewbanin I would love too - BUT - since we are running this through stitch - there's no easy way for me to test. Don't have the bandwidth to spin up a local tap setup.

jcoxdco commented 4 years ago

@drewbanin , I hate to ask as someone not able to contribute to the project, but has anyone found a bit of time to work on the re-auth from RingCentral? I company has about 400 people and currently an hour isn't enough time to get through even 6 hours worth of ring central activity. This would really help my company out.

drewbanin commented 4 years ago

hey @jcoxdco - do you have the ability to test out some code like this on a branch? I'd be happy to send through a PR that... might... fix this if you'd be able to test it out! I no longer have access to a ringcentral api environment

jcoxdco commented 4 years ago

Hey @drewbanin, it would be really difficult for me, but I'm willing to give it a try. I'm just not a python guy. :( I do have the ring central environment to use for testing however.

drewbanin commented 4 years ago

@jcoxdco check it out here: https://github.com/fishtown-analytics/tap-ringcentral/pull/7

This is a pretty simple diff, and the .... ocular ... testing that I just did checks out. Maybe if you're running this code through Stitch, they'd be amenable to setting you up with a test env using this branch?

It does look like singer-io/tap-ringcentral and fishtown-analytics/tap-ringcentral have diverged. Let me send a PR over to the singer-io repo with the same changes for us to discuss... I'll link it here

drewbanin commented 4 years ago

@jcoxdco see here: https://github.com/singer-io/tap-ringcentral/pull/4

Please feel free to comment in that PR accordingly!

wss-chadical commented 4 years ago

@drewbanin Thank you.