search

singer-io / getting-started

This repository is a getting started guide to Singer.
https://singer.io
1.25k stars 148 forks source link

tap-twilio failing on usage_records stream #83

Open lmmuniz opened 3 years ago

lmmuniz commented 3 years ago

Hi,

I have cloned the repo: https://github.com/singer-io/tap-twilio And installed into my venv (Anaconda Python 3.6.5)

I am able to set the config.json to my Twilio account (passing account_sid, auth_token along with start_date and user_agent)

I can run the discovery mode and create the catalog.json:

(venv) (base) C:\Python\environments\tap-twilio\venv>tap-twilio --config config.json --discover > catalog.json
INFO Starting discover
INFO Finished discover
(venv) (base) C:\Python\environments\tap-twilio\venv>

Then I edit the catalog.json to add the attribute "selected": true to usage_records stream:

"stream": "usage_records",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": [
              "account_sid",
              "category",
              "start_date"
            ],
            "forced-replication-method": "INCREMENTAL",
            "valid-replication-keys": [
              "end_date"
            ],
            "inclusion": "available",
            "selected": true
          }

But, when I run the singer-check-tap option in my tap-twilio, it show the following fail autput:

(venv) (base) C:\Python\environments\tap-twilio\venv>tap-twilio --config config.json --catalog catalog.json | singer-check-tap > state.json.tmp

(venv) (base) C:\Python\environments\tap-twilio\venv>tap-twilio --config config.json --catalog catalog.json | singer-check-tap > state.json.tmp
INFO last/currently syncing stream: None
INFO Skipping stream: accounts
INFO Skipping stream: addresses
INFO Skipping stream: dependent_phone_numbers
INFO Skipping stream: applications
INFO Skipping stream: available_phone_number_countries
INFO Skipping stream: available_phone_numbers_local
INFO Skipping stream: available_phone_numbers_mobile
INFO Skipping stream: available_phone_numbers_toll_free
INFO Skipping stream: incoming_phone_numbers
INFO Skipping stream: keys
INFO Skipping stream: calls
INFO Skipping stream: conferences
INFO Skipping stream: conference_participants
INFO Skipping stream: outgoing_caller_ids
INFO Skipping stream: recordings
INFO Skipping stream: transcriptions
INFO Skipping stream: queues
INFO Skipping stream: messages
INFO Skipping stream: message_media
INFO Skipping stream: usage_triggers
INFO Skipping stream: alerts
INFO selected_streams: ['usage_records']
INFO Sync Parent Streams: ['accounts']
INFO Sync Child Streams: ['usage_records']
INFO START Syncing: accounts
INFO stream: accounts, bookmark_field: date_updated, last_datetime: 2019-01-01T00:00:00Z
INFO START Sync for Stream: accounts
INFO URL for Stream accounts: https://api.twilio.com/2010-04-01/Accounts.json
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.21579551696777344, "tags": {"endpoint": "accounts", "http_status_code": 200, "status": "succeeded"}}
INFO START Syncing: usage_records
INFO START Sync for Stream: usage_records, parent_stream: accounts, parent_id: {AccountSID}
INFO No child stream usage_records for parent stream accounts in subresource uris
INFO FINISHED Sync for Stream: usage_records, parent_id: {AccountSID}, total_records: 0
INFO START Sync for Stream: usage_records, parent_stream: accounts, parent_id: {AccountSID}
INFO No child stream usage_records for parent stream accounts in subresource uris
INFO FINISHED Sync for Stream: usage_records, parent_id: {AccountSID}, total_records: 0
INFO START Sync for Stream: usage_records, parent_stream: accounts, parent_id: {AccountSID}
INFO No child stream usage_records for parent stream accounts in subresource uris
INFO FINISHED Sync for Stream: usage_records, parent_id: {AccountSID}, total_records: 0
INFO START Sync for Stream: usage_records, parent_stream: accounts, parent_id: {AccountSID}
INFO No child stream usage_records for parent stream accounts in subresource uris
INFO FINISHED Sync for Stream: usage_records, parent_id: {AccountSID}, total_records: 0
INFO Synced Stream: accounts, page: 1, 0 to 0 of total records: 0
INFO Write state for stream: accounts, value: 2019-01-01T00:00:00Z
INFO FINISHED Syncing: accounts, total_records: 0

(venv) (base) C:\Python\environments\tap-twilio\venv>

My state.json.tmp created shows:

Checking stdin for valid Singer-formatted data The output is valid. It contained 5 messages for 2 streams.

  2 schema messages
  0 record messages
  3 state messages

Details by stream: +---------------+---------+---------+ | stream | records | schemas | +---------------+---------+---------+ | accounts | 0 | 1 | | usage_records | 0 | 1 | +---------------+---------+---------+

My question, is why if shows a message that the usage_records is missing from subresource_uris in Accounts:

INFO No child stream usage_records for parent stream accounts in subresource uris

My accounts requests when I check in Postman shows:

"subresource_uris": { "addresses": "/2010-04-01/Accounts/{AccountSID}/Addresses.json", "conferences": "/2010-04-01/Accounts/{AccountSID}/Conferences.json", "signing_keys": "/2010-04-01/Accounts/{AccountSID}/SigningKeys.json", "transcriptions": "/2010-04-01/Accounts/{AccountSID}/Transcriptions.json", "connect_apps": "/2010-04-01/Accounts/{AccountSID}/ConnectApps.json", "sip": "/2010-04-01/Accounts/{AccountSID}/SIP.json", "authorized_connect_apps": "/2010-04-01/Accounts/{AccountSID}/AuthorizedConnectApps.json", "usage": "/2010-04-01/Accounts/{AccountSID}/Usage.json", "keys": "/2010-04-01/Accounts/{AccountSID}/Keys.json", "applications": "/2010-04-01/{AccountSID}/Applications.json", "recordings": "/2010-04-01/Accounts/{AccountSID}/Recordings.json", "short_codes": "/2010-04-01/Accounts/{AccountSID}/SMS/ShortCodes.json", "calls": "/2010-04-01/Accounts/{AccountSID}/Calls.json", "notifications": "/2010-04-01/Accounts/{AccountSID}/Notifications.json", "incoming_phone_numbers": "/2010-04-01/Accounts/{AccountSID}/IncomingPhoneNumbers.json", "queues": "/2010-04-01/Accounts/{AccountSID}/Queues.json", "messages": "/2010-04-01/Accounts/{AccountSID}/Messages.json", "outgoing_caller_ids": "/2010-04-01/Accounts/{AccountSID}/OutgoingCallerIds.json", "available_phone_numbers": "/2010-04-01/Accounts/{AccountSID}/AvailablePhoneNumbers.json", "balance": "/2010-04-01/Accounts/{AccountSID}/Balance.json" }

And when i open usage request in Postman:

{ "subresource_uris": { "records": "/2010-04-01/Accounts/{AccountSID}/Usage/Records.json", "triggers": "/2010-04-01/Accounts/{AccountSID}/Usage/Triggers.json" } }

What is missing in the tap-twilio repo to make this API request to work?

Any help is kindly thankful.

Leonardo Muniz