search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.53k stars 4k forks source link

[source-amazon-seller-partner] setup source for Vendors fails #33929

Closed zata666 closed 7 months ago

zata666 commented 8 months ago

Connector Name

source-amazon-seller-partner

Connector Version

3.0.1

What step the error happened?

Configuring a new connector

Relevant information

I'm not sure if this is the problem but for Vendors this shouldn't be checked: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://sellingpartnerapi-eu.amazon.com/orders/v0/orders?LastUpdatedAfter=2022-01-04T13%3A18%3A38Z&MaxResultsPerPage=100&MarketplaceIds=A1PA6795UKMFR9

image

For Seller account type everything works fine.

Relevant log output

2024-01-04 13:18:37 platform > Docker volume job log path: /tmp/workspace/ba35d158-9730-43d0-b373-0ce9ed1e42f9/0/logs.log
2024-01-04 13:18:37 platform > Executing worker wrapper. Airbyte version: 0.50.40
2024-01-04 13:18:37 platform > Attempt 0 to save workflow id for cancellation
2024-01-04 13:18:37 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-04 13:18:37 platform > 
2024-01-04 13:18:37 platform > ----- START check-orchestrator -----
2024-01-04 13:18:37 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-04 13:18:37 platform > 
2024-01-04 13:18:37 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-04 13:18:37 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-04 13:18:37 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-04 13:18:37 platform > Checking if airbyte/source-amazon-seller-partner:3.0.1 exists...
2024-01-04 13:18:37 platform > airbyte/source-amazon-seller-partner:3.0.1 was found locally.
2024-01-04 13:18:37 platform > Creating docker container = source-amazon-seller-partner-check-ba35d158-9730-43d0-b373-0ce9ed1e42f9-0-ceooh with resources io.airbyte.config.ResourceRequirements@e5b12f1[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-01-04 13:18:37 platform > Preparing command: docker run --rm --init -i -w /data/ba35d158-9730-43d0-b373-0ce9ed1e42f9/0 --log-driver none --name source-amazon-seller-partner-check-ba35d158-9730-43d0-b373-0ce9ed1e42f9-0-ceooh --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-amazon-seller-partner:3.0.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.40 -e WORKER_JOB_ID=ba35d158-9730-43d0-b373-0ce9ed1e42f9 airbyte/source-amazon-seller-partner:3.0.1 check --config source_config.json
2024-01-04 13:18:37 platform > Reading messages from protocol version 0.2.0
2024-01-04 13:18:39 platform > {
  "errors": [
    {
      "code": "Unauthorized",
      "message": "Access to requested resource is denied.",
      "details": ""
    }
  ]
}
2024-01-04 13:19:10 platform > None
Traceback (most recent call last):
  File "/airbyte/integration_code/source_amazon_seller_partner/source.py", line 116, in check_connection
    next(orders_stream.read_records(sync_mode=SyncMode.full_refresh))
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 482, in read_records
    yield from self._read_pages(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 498, in _read_pages
    request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 524, in _fetch_next_page
    response = self._send_request(request, request_kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 422, in _send_request
    return backoff_handler(user_backoff_handler)(request, request_kwargs)
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 381, in _send
    raise exc
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 378, in _send
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://sellingpartnerapi-eu.amazon.com/orders/v0/orders?LastUpdatedAfter=2022-01-04T13%3A18%3A38Z&MaxResultsPerPage=100&MarketplaceIds=A1PA6795UKMFR9

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 16, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 213, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 113, in run
    yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.check(source_spec, config))
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 137, in check
    check_result = self.source.check(self.logger, config)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 84, in check
    check_succeeded, error = self.check_connection(logger, config)
  File "/airbyte/integration_code/source_amazon_seller_partner/source.py", line 127, in check_connection
    next(stream_to_check.read_records(sync_mode=SyncMode.full_refresh))
  File "/airbyte/integration_code/source_amazon_seller_partner/streams.py", line 368, in read_records
    raise AirbyteTracedException(message=f"The report for stream '{self.name}' was not created - skip reading")
airbyte_cdk.utils.traced_exception.AirbyteTracedException: None
2024-01-04 13:19:10 platform > Check connection job subprocess finished with exit code 1
2024-01-04 13:19:10 platform > 
2024-01-04 13:19:10 platform > ----- END check-orchestrator -----
2024-01-04 13:19:10 platform >

Contribute

zata666 commented 7 months ago

In 3.2 version still the same error - log below check_connection_source-failure-8f3613f5-0c86-4ff6-addc-5309539fac2a.txt

tolik0 commented 7 months ago

@zata666 Hi, it seems the issue is related to a Fatal error in the VendorSalesReport. We've noticed that similar issues have been reported on the Amazon GitHub repository, but, unfortunately, none have yet been resolved. Could you please recreate your connection in Airbyte Cloud using your Vendor account? Doing so will enable us to perform additional testing and gain a deeper understanding of the problem.

zata666 commented 7 months ago

In cloud I'm also getting error while creating connection :(

check_connection_source-failure-101533dc-33e4-431f-ace9-93a3b169ff4b.txt

tkemb commented 7 months ago

We have the same issue with Airbyte Self-Managed. trying to connect a Amazon Vendor account, but can't configure the source with the error GET_VENDOR_SALES_REPORT.

tolik0 commented 7 months ago

@zata666, @tkemb Hi, I merged the PR with the fix for the check command for the Vendor account type. Could you please check if you can now create a connection?

tkemb commented 7 months ago

Tested the PR, the error is fixed, but we get another error now, KeyError('dataStartTime') , error log: check_connection_source-failure-d989f41e-9f91-4706-8206-4f102f444449.txt

zata666 commented 7 months ago

Connection created successfully - thank You :)

zata666 commented 7 months ago

Hmmm, in version 3.3.2 self hosted I added successfully one source for vendor type account but now after upgrading to 3.4.0 (and also downgrading back to 3.3.2) I'm getting same error as tkemb check_connection_source-failure-d989f41e-9f91-4706-8206-4f102f444449.txt

In cloud version it still works fine

tolik0 commented 7 months ago

@zata666 Hello, please update to version 3.5.0, which includes additional fixes for this issue.

zata666 commented 7 months ago

everything works fine now - thanks.