search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.2k stars 4.14k forks source link

Source AppStore - Skipping dates known to exist #23691

Closed daytenjs closed 7 months ago

daytenjs commented 1 year ago

Environment

Current Behavior

I get like 8 print outs of the following:

Invalid vendor number specified. Try again. 2023-03-02 19:10:52 source > An unexpected error occurred on the server side. If this issue continues, contact us at https://developer.apple.com/contact/.

I also get

2023-03-02 19:10:52 ERROR c.n.s.DateTimeValidator(tryParse):82 - Invalid date: Text '10/17/2022' could not be parsed at index 0

despite definitely having data for that date.

The job ends up skipping like half the dates (that I know are in fact there), and I'm getting massively incomplete data, with entire weeks skipped.

Expected Behavior

I've already gotten all the data before, suddenly it stopped syncing (with no user changes) latest dates, on a reset and re-sync these issues started. The "invalid vendor id" issue has been around since I started using the connector. The vendor id works fine when I use python to call the Appstore API.

Logs

3878c2c0_44b4_428b_b671_21d8ed40b50d_logs_127_txt.txt

Steps to Reproduce

  1. Have an appstore connection
  2. sync, reset, re-sync
daytenjs commented 1 year ago

Moreover, when I "refresh source schema" in the replication tab of the connection settings, it tells me it removes the table (that I have successfully imported many times)

image
daytenjs commented 1 year ago

This connector would also greatly benefit from a "full overwrite" setting... so that if it skips dates, it will pick them up on later syncs...

sajarin commented 1 year ago

Thanks @daytenjs, I triaged and added this ticket to the team's backlog!

pushchris commented 1 year ago

@sajarin this is because the source is using version 1_2 for most of those reports which is no longer support by App Store Connect. Effectively the source is mostly broken and wont pull those vital reports anymore. Should be a three character change to get the source to pull using the correct version key.

For more information: https://developer.apple.com/documentation/appstoreconnectapi/download_sales_and_trends_reports#3022173

References in the Airbyte source: https://github.com/airbytehq/airbyte/blob/2da66e9cf14976b243e330ea39b7469a3d11813f/airbyte-integrations/connectors/source-appstore-singer/source_appstore_singer/source.py#LL40C13-L50C14

marcosmarxm commented 7 months ago

The App Store connector has been archived and won't get updates in the future. We're closing this issue to focus on features and connectors in our roadmap. Thanks for understanding! 🚀