Joe-Heffer-MPE commented 5 years ago

Description of change

Added a new sync schema and sync function to retrieve all Taboola accounts which the authenticated user has access to.

The automatic code formatter on my IDE (PyCharm) applied some formatting to the other bits of code which are practically unchanged otherwise.

Manual QA steps

Risks

Rollback steps

revert this branch

Joe-Heffer-MPE commented 5 years ago

This PR will also sync campaigns/campaign performance across all accounts

gbsilv commented 4 years ago

Hey, we are in need of these feature, could someone check it out and merge, please? Thanks!

nmccormick-bombas commented 4 years ago

@Joe-Heffer-MPE I approved the PR - are you able to merge? I'm in need of this fix as well.

rumeau commented 3 years ago

Hello, I wanted to implement this feature. But i've got

A record for stream accountswas encountered before a corresponding schema

Error when extracting data and sending it to CSV

This is my error log.

meltano | No state was found, complete import.
tap-taboola-jp | INFO Starting sync.
tap-taboola-jp | INFO Generating new token with password auth
target-csv | INFO Sending version information to singer.io. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
tap-taboola-jp | INFO Got response code: 400
tap-taboola-jp | INFO Retrying with client credentials authentication.
tap-taboola-jp | INFO Got response code: 200
tap-taboola-jp | INFO Got an access token.
tap-taboola-jp | INFO Making request: GET https://backstage.taboola.com/backstage/api/1.0/token-details/ {}
tap-taboola-jp | INFO Got response code: 200
tap-taboola-jp | INFO Verified account access via token details endpoint.
tap-taboola-jp | INFO Making request: GET https://backstage.taboola.com/backstage/api/1.0/users/current/allowed-accounts/ {}
tap-taboola-jp | INFO Got response code: 200
tap-taboola-jp | INFO Synced 13 accounts.
target-csv | Traceback (most recent call last):
target-csv | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/loaders/target-csv/venv/bin/target-csv", line 8, in <module>
target-csv | sys.exit(main())
target-csv | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/loaders/target-csv/venv/lib/python3.8/site-packages/target_csv.py", line 141, in main
target-csv | state = persist_messages(config.get('delimiter', ','),
target-csv | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/loaders/target-csv/venv/lib/python3.8/site-packages/target_csv.py", line 56, in persist_messages
target-csv | raise Exception("A record for stream {}"
target-csv | Exception: A record for stream accountswas encountered before a corresponding schema
tap-taboola-jp | INFO Done syncing accounts.
tap-taboola-jp | INFO Making request: GET https://backstage.taboola.com/backstage/api/1.0/****************/campaigns/ {}
tap-taboola-jp | INFO Got response code: 200
tap-taboola-jp | INFO Synced 0 campaigns.
tap-taboola-jp | INFO Done syncing campaigns.
tap-taboola-jp | INFO Making request: GET https://backstage.taboola.com/backstage/api/1.0/****************/reports/campaign-summary/dimensions/campaign_day_breakdown {'start_date': '2020-01-01', 'end_date': datetime.date(2020, 12, 5)}
tap-taboola-jp | INFO Got response code: 200
tap-taboola-jp | INFO Got 120 campaign performance records.
tap-taboola-jp | CRITICAL [Errno 32] Broken pipe
tap-taboola-jp | Traceback (most recent call last):
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/bin/tap-taboola", line 8, in <module>
tap-taboola-jp | sys.exit(main())
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/tap_taboola/__init__.py", line 394, in main
tap-taboola-jp | raise exc
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/tap_taboola/__init__.py", line 391, in main
tap-taboola-jp | main_impl()
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/tap_taboola/__init__.py", line 383, in main_impl
tap-taboola-jp | do_sync(args)
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/tap_taboola/__init__.py", line 368, in do_sync
tap-taboola-jp | sync_campaign_performance(config, state, access_token,
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/tap_taboola/__init__.py", line 169, in sync_campaign_performance
tap-taboola-jp | singer.write_record('campaign_performance',
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/singer/messages.py", line 216, in write_record
tap-taboola-jp | write_message(RecordMessage(stream=(stream_alias or stream_name),
tap-taboola-jp | File "/home/ubuntu/meltano-projects/taboola2bigquery/.meltano/extractors/tap-taboola-jp/venv/lib/python3.8/site-packages/singer/messages.py", line 208, in write_message
tap-taboola-jp | sys.stdout.flush()
tap-taboola-jp | BrokenPipeError: [Errno 32] Broken pipe
tap-taboola-jp | Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
tap-taboola-jp | BrokenPipeError: [Errno 32] Broken pipe
meltano | Extraction failed (120): BrokenPipeError: [Errno 32] Broken pipe
meltano | Loading failed (1): Exception: A record for stream accountswas encountered before a corresponding schema
meltano | ELT could not be completed: Tap and target failed

Can you help me debug this. As my knowledge on Python is limited.

Thanks