search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.04k stars 4.11k forks source link

BUG - Amazon Ads connector doesn't pull data #11011

Closed ttakizawa closed 2 years ago

ttakizawa commented 2 years ago
## Environment - **Airbyte version**: 0.35.45-alpha - **OS Version / Instance**: macOS - **Deployment**: Docker - **Source Connector and version**: Amazon Ads 0.1.3 - **Destination Connector and version**: BigQuery 0.6.10 - **Severity**: High - **Step where error happened**: Sync job ## Current Behavior Getting error in sync job. Settings is following Source Settings - Scope: advertising::campaign_management - AmazonAdsRegion: SANDBOX - Start Date: 2022-03-01 Connection Settings - Streams: sponsored_products_report_stream ## Expected Behavior pull data from Amazon Ads API ## Logs
LOG ``` 2022-03-08 02:16:14 INFO i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.35.45-alpha 2022-03-08 02:16:14 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/15/2/logs.log 2022-03-08 02:16:14 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.45-alpha 2022-03-08 02:16:14 INFO i.a.w.DefaultReplicationWorker(run):103 - start sync worker. job id: 15 attempt id: 2 2022-03-08 02:16:14 INFO i.a.w.DefaultReplicationWorker(run):115 - configured sync modes: {null.sponsored_products_report_stream=full_refresh - append} 2022-03-08 02:16:14 INFO i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination... 2022-03-08 02:16:14 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/destination-bigquery:0.6.10 exists... 2022-03-08 02:16:14 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/destination-bigquery:0.6.10 was found locally. 2022-03-08 02:16:14 INFO i.a.w.p.DockerProcessFactory(create):104 - Creating docker job ID: 15 2022-03-08 02:16:14 INFO i.a.w.p.DockerProcessFactory(create):155 - Preparing command: docker run --rm --init -i -w /data/15/2 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/destination-bigquery:0.6.10 write --config destination_config.json --catalog destination_catalog.json 2022-03-08 02:16:14 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-amazon-ads:0.1.3 exists... 2022-03-08 02:16:14 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-amazon-ads:0.1.3 was found locally. 2022-03-08 02:16:14 INFO i.a.w.p.DockerProcessFactory(create):104 - Creating docker job ID: 15 2022-03-08 02:16:14 INFO i.a.w.p.DockerProcessFactory(create):155 - Preparing command: docker run --rm --init -i -w /data/15/2 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/source-amazon-ads:0.1.3 read --config source_config.json --catalog source_catalog.json --state input_state.json 2022-03-08 02:16:14 INFO i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):338 - Destination output thread started. 2022-03-08 02:16:14 INFO i.a.w.DefaultReplicationWorker(run):157 - Waiting for source and destination threads to complete. 2022-03-08 02:16:14 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):278 - Replication thread started. 2022-03-08 02:16:15 destination > SLF4J: Class path contains multiple SLF4J bindings. 2022-03-08 02:16:15 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2022-03-08 02:16:15 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2022-03-08 02:16:15 destination > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2022-03-08 02:16:15 destination > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2022-03-08 02:16:15 source > Starting syncing SourceAmazonAds 2022-03-08 02:16:16 destination > 2022-03-08 02:16:16 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 INFO i.a.i.b.IntegrationRunner(run):88 - Sentry transaction event: 9ab31a4336cd4c4cbc94b8b48efde362 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 INFO i.a.i.b.IntegrationRunner(runInternal):107 - Command: WRITE 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 INFO i.a.i.b.IntegrationRunner(runInternal):108 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-03-08 02:16:17 destination > 2022-03-08 02:16:17 INFO i.a.i.d.b.BigQueryUtils(getLoadingMethod):283 - Selected loading method is set to: STANDARD 2022-03-08 02:16:18 source > Syncing stream: sponsored_products_report_stream 2022-03-08 02:16:19 source > Initiating report generation for 75718738545707 profile with campaigns type for 20220228 date 2022-03-08 02:16:19 source > Backing off _init_reports(...) for 0.4s (source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"WDF32TXJ7V329WN6KHE3"}) 2022-03-08 02:16:19 destination > 2022-03-08 02:16:19 INFO i.a.i.d.b.BigQueryUtils(createPartitionedTable):123 - Partitioned Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte, tableId=_airbyte_tmp_gcj_sponsored_products_report_stream}} created successfully 2022-03-08 02:16:20 source > Initiating report generation for 75718738545707 profile with campaigns type for 20220228 date 2022-03-08 02:16:20 source > Backing off _init_reports(...) for 1.8s (source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"VYX5QCF6ZJQTHJ43CSXF"}) 2022-03-08 02:16:22 source > Initiating report generation for 75718738545707 profile with campaigns type for 20220228 date 2022-03-08 02:16:22 source > Backing off _init_reports(...) for 3.1s (source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"34BQNHF89D41PQSDP1R6"}) 2022-03-08 02:16:25 source > Initiating report generation for 75718738545707 profile with campaigns type for 20220228 date 2022-03-08 02:16:25 source > Backing off _init_reports(...) for 7.8s (source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"A5X8RZCZ4EDNP8BT0K82"}) 2022-03-08 02:16:33 source > Initiating report generation for 75718738545707 profile with campaigns type for 20220228 date 2022-03-08 02:16:33 source > Giving up _init_reports(...) after 5 tries (source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"HSV6F9W2B42P9EQNR2XN"}) 2022-03-08 02:16:33 source > /usr/local/lib/python3.7/site-packages/pendulum/pendulum.py:2086: PendulumDeprecationWarning: The add_timedelta() method will be removed in version 2.0. 2022-03-08 02:16:33 source > Encountered an exception while reading stream SourceAmazonAds Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 108, in read internal_config=internal_config, File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 141, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 213, in _read_full_refresh for record in records: File "/airbyte/integration_code/source_amazon_ads/streams/report_streams/report_streams.py", line 142, in read_records report_infos = self._init_and_try_read_records(report_date) File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry ret = target(*args, **kwargs) File "/airbyte/integration_code/source_amazon_ads/streams/report_streams/report_streams.py", line 159, in _init_and_try_read_records report_infos = self._init_reports(report_date) File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry ret = target(*args, **kwargs) File "/airbyte/integration_code/source_amazon_ads/streams/report_streams/report_streams.py", line 345, in _init_reports f"Unexpected HTTP status code {response.status_code} when registering {record_type}, {type(self).__name__} for {profile.profileId} profile: {response.text}" source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"HSV6F9W2B42P9EQNR2XN"} 2022-03-08 02:16:33 source > return self.add_timedelta(other) 2022-03-08 02:16:33 source > Finished syncing SourceAmazonAds 2022-03-08 02:16:33 source > SourceAmazonAds runtimes: 2022-03-08 02:16:33 source > Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"HSV6F9W2B42P9EQNR2XN"} Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in launch(source, sys.argv[1:]) File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 116, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 107, in run for message in generator: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 112, in read raise e File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 108, in read internal_config=internal_config, File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 141, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 213, in _read_full_refresh for record in records: File "/airbyte/integration_code/source_amazon_ads/streams/report_streams/report_streams.py", line 142, in read_records report_infos = self._init_and_try_read_records(report_date) File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry ret = target(*args, **kwargs) File "/airbyte/integration_code/source_amazon_ads/streams/report_streams/report_streams.py", line 159, in _init_and_try_read_records report_infos = self._init_reports(report_date) File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry ret = target(*args, **kwargs) File "/airbyte/integration_code/source_amazon_ads/streams/report_streams/report_streams.py", line 345, in _init_reports f"Unexpected HTTP status code {response.status_code} when registering {record_type}, {type(self).__name__} for {profile.profileId} profile: {response.text}" source_amazon_ads.streams.report_streams.report_streams.ReportInitFailure: Unexpected HTTP status code 400 when registering campaigns, SponsoredProductsReportStream for 75718738545707 profile: {"code":"400","details":"Unsupported metric bidPlus","requestId":"HSV6F9W2B42P9EQNR2XN"} 2022-03-08 02:16:34 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):304 - Total records read: 0 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):65 - Airbyte message consumer: succeeded. 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.d.b.BigQueryRecordConsumer(close):55 - Started closing all connections 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.d.b.u.AbstractBigQueryUploader(close):76 - Field fails during format : 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.d.b.f.BigQueryRecordFormatter(printAndCleanFieldFails):70 - No field fails during record format. 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.d.b.u.AbstractBigQueryUploader(close):79 - Closing connector:AbstractBigQueryUploader{table=_airbyte_raw_sponsored_products_report_stream, tmpTable=_airbyte_tmp_gcj_sponsored_products_report_stream, syncMode=WRITE_APPEND, writer=class io.airbyte.integrations.destination.bigquery.writer.BigQueryTableWriter, recordFormatter=class io.airbyte.integrations.destination.bigquery.formatter.DefaultBigQueryRecordFormatter} 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.d.b.u.AbstractBigQueryUploader(uploadData):96 - Uploading data from the tmp table _airbyte_tmp_gcj_sponsored_products_report_stream to the source table _airbyte_raw_sponsored_products_report_stream. 2022-03-08 02:16:34 destination > 2022-03-08 02:16:34 INFO i.a.i.d.b.u.AbstractBigQueryUploader(uploadDataToTableFromTmpTable):121 - Replication finished with no explicit errors. Copying data from tmp tables to permanent 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.d.b.u.AbstractBigQueryUploader(copyTable):187 - successfully copied table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte, tableId=_airbyte_tmp_gcj_sponsored_products_report_stream}} to table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte, tableId=_airbyte_raw_sponsored_products_report_stream}} 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.d.b.u.AbstractBigQueryUploader(uploadData):98 - Data is successfully loaded to the source table _airbyte_raw_sponsored_products_report_stream! 2022-03-08 02:16:36 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. errors: $: null found, object expected 2022-03-08 02:16:36 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.d.b.u.AbstractBigQueryUploader(uploadData):100 - Final state message is accepted. 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.d.b.u.AbstractBigQueryUploader(dropTmpTable):112 - Removing tmp tables... 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.d.b.u.AbstractBigQueryUploader(dropTmpTable):114 - Finishing destination process...completed 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.d.b.u.AbstractBigQueryUploader(close):86 - Closed connector:AbstractBigQueryUploader{table=_airbyte_raw_sponsored_products_report_stream, tmpTable=_airbyte_tmp_gcj_sponsored_products_report_stream, syncMode=WRITE_APPEND, writer=class io.airbyte.integrations.destination.bigquery.writer.BigQueryTableWriter, recordFormatter=class io.airbyte.integrations.destination.bigquery.formatter.DefaultBigQueryRecordFormatter} 2022-03-08 02:16:36 destination > 2022-03-08 02:16:36 INFO i.a.i.b.IntegrationRunner(runInternal):154 - Completed integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2022-03-08 02:16:36 ERROR i.a.w.DefaultReplicationWorker(run):168 - Sync worker failed. java.util.concurrent.ExecutionException: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:161) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:56) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at java.lang.Thread.run(Thread.java:833) [?:?] Suppressed: io.airbyte.workers.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:125) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:56) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1 at io.airbyte.workers.DefaultReplicationWorker.lambda$getReplicationRunnable$5(DefaultReplicationWorker.java:311) ~[io.airbyte-airbyte-workers-0.35.45-alpha.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] ... 1 more 2022-03-08 02:16:36 INFO i.a.w.DefaultReplicationWorker(run):227 - sync summary: io.airbyte.config.ReplicationAttemptSummary@6728ae89[status=failed,recordsSynced=0,bytesSynced=0,startTime=1646705774288,endTime=1646705796718,totalStats=io.airbyte.config.SyncStats@3b834ea8[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]] 2022-03-08 02:16:36 INFO i.a.w.DefaultReplicationWorker(run):249 - Source did not output any state messages 2022-03-08 02:16:36 WARN i.a.w.DefaultReplicationWorker(run):257 - State capture: No new state, falling back on input state: io.airbyte.config.State@6726552a[state={}] 2022-03-08 02:16:36 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling... 2022-03-08 02:16:36 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$1):147 - sync summary: io.airbyte.config.StandardSyncOutput@56f5f652[standardSyncSummary=io.airbyte.config.StandardSyncSummary@a7a06f4[status=failed,recordsSynced=0,bytesSynced=0,startTime=1646705774288,endTime=1646705796718,totalStats=io.airbyte.config.SyncStats@3b834ea8[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]],state=io.airbyte.config.State@6726552a[state={}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@2d85c760[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@1f94648e[stream=io.airbyte.protocol.models.AirbyteStream@662a5648[name=sponsored_products_report_stream,jsonSchema={"type":"object","properties":{"metric":{"type":"object","properties":{"sku":{"type":["null","string"]},"asin":{"type":["null","string"]},"cost":{"type":["null","string"]},"clicks":{"type":["null","string"]},"bidPlus":{"type":["null","string"]},"currency":{"type":["null","string"]},"targetId":{"type":["null","string"]},"adGroupId":{"type":["null","string"]},"keywordId":{"type":["null","string"]},"matchType":{"type":["null","string"]},"otherAsin":{"type":["null","string"]},"campaignId":{"type":["null","string"]},"adGroupName":{"type":["null","string"]},"impressions":{"type":["null","string"]},"keywordText":{"type":["null","string"]},"campaignName":{"type":["null","string"]},"targetingText":{"type":["null","string"]},"targetingType":{"type":["null","string"]},"campaignBudget":{"type":["null","string"]},"campaignStatus":{"type":["null","string"]},"attributedSales1d":{"type":["null","string"]},"attributedSales7d":{"type":["null","string"]},"attributedSales14d":{"type":["null","string"]},"attributedSales30d":{"type":["null","string"]},"targetingExpression":{"type":["null","string"]},"applicableBudgetRuleId":{"type":["null","string"]},"attributedConversions1d":{"type":["null","string"]},"attributedConversions7d":{"type":["null","string"]},"campaignRuleBasedBudget":{"type":["null","string"]},"applicableBudgetRuleName":{"type":["null","string"]},"attributedConversions14d":{"type":["null","string"]},"attributedConversions30d":{"type":["null","string"]},"attributedSales1dSameSKU":{"type":["null","string"]},"attributedSales7dSameSKU":{"type":["null","string"]},"attributedUnitsOrdered1d":{"type":["null","string"]},"attributedUnitsOrdered7d":{"type":["null","string"]},"attributedSales14dSameSKU":{"type":["null","string"]},"attributedSales1dOtherSKU":{"type":["null","string"]},"attributedSales30dSameSKU":{"type":["null","string"]},"attributedSales7dOtherSKU":{"type":["null","string"]},"attributedUnitsOrdered14d":{"type":["null","string"]},"attributedUnitsOrdered30d":{"type":["null","string"]},"attributedSales14dOtherSKU":{"type":["null","string"]},"attributedSales30dOtherSKU":{"type":["null","string"]},"attributedConversions1dSameSKU":{"type":["null","string"]},"attributedConversions7dSameSKU":{"type":["null","string"]},"attributedConversions14dSameSKU":{"type":["null","string"]},"attributedConversions30dSameSKU":{"type":["null","string"]},"attributedUnitsOrdered1dSameSKU":{"type":["null","string"]},"attributedUnitsOrdered7dSameSKU":{"type":["null","string"]},"attributedUnitsOrdered14dSameSKU":{"type":["null","string"]},"attributedUnitsOrdered1dOtherSKU":{"type":["null","string"]},"attributedUnitsOrdered30dSameSKU":{"type":["null","string"]},"attributedUnitsOrdered7dOtherSKU":{"type":["null","string"]},"attributedUnitsOrdered14dOtherSKU":{"type":["null","string"]},"attributedUnitsOrdered30dOtherSKU":{"type":["null","string"]}}},"profileId":{"type":"integer"},"recordType":{"type":"string"},"reportDate":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[reportDate],sourceDefinedPrimaryKey=[],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[reportDate],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[io.airbyte.config.FailureReason@3cbdb1df[failureOrigin=source,failureType=,internalMessage=io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@3f80dfef[additionalProperties={attemptNumber=2, jobId=15}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1 at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1 at io.airbyte.workers.DefaultReplicationWorker.lambda$getReplicationRunnable$5(DefaultReplicationWorker.java:311) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 3 more ,retryable=,timestamp=1646705794021]]] 2022-03-08 02:16:36 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating... 2022-03-08 02:16:36 INFO i.a.c.p.ConfigRepository(updateConnectionState):563 - Updating connection 4687d316-9482-4680-9c7e-9d0b06a7a210 state: io.airbyte.config.State@66c28ed8[state={}] 2022-03-08 02:16:36 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/15/2/logs.log 2022-03-08 02:16:36 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.45-alpha 2022-03-08 02:16:36 INFO i.a.w.DefaultNormalizationWorker(run):46 - Running normalization. 2022-03-08 02:16:36 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):122 - Running with normalization version: airbyte/normalization:0.1.68 2022-03-08 02:16:36 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.68 exists... 2022-03-08 02:16:37 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/normalization:0.1.68 was found locally. 2022-03-08 02:16:37 INFO i.a.w.p.DockerProcessFactory(create):104 - Creating docker job ID: 15 2022-03-08 02:16:37 INFO i.a.w.p.DockerProcessFactory(create):155 - Preparing command: docker run --rm --init -i -w /data/15/2/normalize --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/normalization:0.1.68 run --integration-type bigquery --config destination_config.json --catalog destination_catalog.json 2022-03-08 02:16:37 normalization > Running: transform-config --config destination_config.json --integration-type bigquery --out /data/15/2/normalize 2022-03-08 02:16:37 normalization > Namespace(config='destination_config.json', integration_type=, out='/data/15/2/normalize') 2022-03-08 02:16:37 normalization > transform_bigquery 2022-03-08 02:16:37 normalization > Running: transform-catalog --integration-type bigquery --profile-config-dir /data/15/2/normalize --catalog destination_catalog.json --out /data/15/2/normalize/models/generated/ --json-column _airbyte_data 2022-03-08 02:16:37 normalization > Processing destination_catalog.json... 2022-03-08 02:16:37 normalization > Generating airbyte_ctes/airbyte/sponsored_products_report_stream_ab1.sql from sponsored_products_report_stream 2022-03-08 02:16:37 normalization > Generating airbyte_ctes/airbyte/sponsored_products_report_stream_ab2.sql from sponsored_products_report_stream 2022-03-08 02:16:37 normalization > Generating airbyte_ctes/airbyte/sponsored_products_report_stream_ab3.sql from sponsored_products_report_stream 2022-03-08 02:16:37 normalization > Generating airbyte_incremental/airbyte/sponsored_products_report_stream.sql from sponsored_products_report_stream 2022-03-08 02:16:37 normalization > Generating airbyte_ctes/airbyte/sponsored_products_report_stream_metric_ab1.sql from sponsored_products_report_stream/metric 2022-03-08 02:16:37 normalization > Generating airbyte_ctes/airbyte/sponsored_products_report_stream_metric_ab2.sql from sponsored_products_report_stream/metric 2022-03-08 02:16:37 normalization > Generating airbyte_ctes/airbyte/sponsored_products_report_stream_metric_ab3.sql from sponsored_products_report_stream/metric 2022-03-08 02:16:37 normalization > Generating airbyte_incremental/airbyte/sponsored_products_report_stream_metric.sql from sponsored_products_report_stream/metric 2022-03-08 02:16:37 normalization > detected no config file for ssh, assuming ssh is off. 2022-03-08 02:16:40 normalization > Running with dbt=0.21.1 2022-03-08 02:16:40 normalization > Unable to do partial parsing because ../build/partial_parse.msgpack not found 2022-03-08 02:16:42 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources. 2022-03-08 02:16:42 normalization > There are 2 unused configuration paths: 2022-03-08 02:16:42 normalization > - models.airbyte_utils.generated.airbyte_views 2022-03-08 02:16:42 normalization > - models.airbyte_utils.generated.airbyte_tables 2022-03-08 02:16:42 normalization > 2022-03-08 02:16:42 normalization > Found 8 models, 0 tests, 0 snapshots, 0 analyses, 516 macros, 0 operations, 0 seed files, 1 source, 0 exposures 2022-03-08 02:16:42 normalization > 2022-03-08 02:16:44 normalization > 02:16:44 | Concurrency: 8 threads (target='prod') 2022-03-08 02:16:44 normalization > 02:16:44 | 2022-03-08 02:16:46 normalization > 02:16:46 | 1 of 2 START incremental model airbyte.sponsored_products_report_stream...................................... [RUN] 2022-03-08 02:16:52 normalization > 02:16:52 | 1 of 2 OK created incremental model airbyte.sponsored_products_report_stream................................. [MERGE (0.0 rows, 0.0 Bytes processed) in 5.78s] 2022-03-08 02:16:54 normalization > 02:16:54 | 2 of 2 START incremental model airbyte.sponsored_products_report_stream_metric............................... [RUN] 2022-03-08 02:17:00 normalization > 02:17:00 | 2 of 2 OK created incremental model airbyte.sponsored_products_report_stream_metric.......................... [MERGE (0.0 rows, 0.0 Bytes processed) in 5.44s] 2022-03-08 02:17:00 normalization > 02:17:00 | 2022-03-08 02:17:00 normalization > 02:17:00 | Finished running 2 incremental models in 17.28s. 2022-03-08 02:17:00 normalization > 2022-03-08 02:17:00 normalization > Completed successfully 2022-03-08 02:17:00 normalization > 2022-03-08 02:17:00 normalization > Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 2022-03-08 02:17:00 INFO i.a.w.DefaultNormalizationWorker(run):69 - Normalization executed in 23 seconds. 2022-03-08 02:17:00 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling... 2022-03-08 02:17:00 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating... ```
## Steps to Reproduce 1. connect Amazon Ads to BigQuery 2. add sponsored_products_report_stream sources - sync method Full Refresh | Append 3. run sync manually
grubberr commented 2 years ago

According to docs this metric bidPlus CAN be used for: Ad type: Sponsored Products Record types: campaigns url: https://advertising-api-test.amazon.com/v2/sp/campaigns/report

It's not clear why Amazon Ads API response with error: 

{"code":"400","details":"Unsupported metric bidPlus","requestId":"486PQPQZYJ28VPDMZBX1"}

Probably we need to create jira issue here https://amzn-clicks.atlassian.net/servicedesk/customer/portal/2/group/2 to continue to investigate

What is also interesting if we use really non-existent metric notfound we got another error message:

{"code":"422","details":"Unrecognized metric: notfound\\n at [Source:","requestId":"HSQ4CXY4VAG87YJCS9CN"}
grubberr commented 2 years ago

For now all looks like amazon SANDBOX pretty buggy and a lot of things does not work on SANDBOX region. Stream sponsored_products_report_stream only works under "Test account" https://advertising.amazon.com/API/docs/en-us/test-accounts/overview

@ttakizawa if it ok with you, please close this issue

grubberr commented 2 years ago

@ttakizawa can we close this issue ?