search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.97k stars 3.85k forks source link

Source Facebook Marketing: Investigate Discrepancies in Data and Consider Switching from Async Reports to GET Requests #31860

Open tolik0 opened 9 months ago

tolik0 commented 9 months ago

Connector Name

source-facebook-marketing

Connector Version

1.1.17

What step the error happened?

During the sync

Relevant information

Problem:

We've identified discrepancies in the data retrieved by the Airbyte Facebook Marketing Connector, specifically within the ads_insights stream. The action_values column is null in both raw and normalized tables, impacting the revenue and orders columns. Another user highlighted on Slack that the arrays actions and action_values in the ads_insights stream no longer encompass offsite_conversion types, hindering the extraction of crucial metrics.

A customer also reported similar inconsistencies in Issue #31787. The error was pinpointed to the method of data retrieval from the Facebook API through Ad Report Run. This method aggregates pixel data values, obscuring specific metric details. The customer's investigation suggests that the error arises when data is requested with the time_increment field set to 1, leading to aggregated values that make it challenging to discern specific metrics.

Previous Investigation:

An in-depth investigation was undertaken, which included reverting to prior versions of the connector and attempting to exclude potentially conflicting fields. However, the root cause remains elusive. Three issues on the Facebook platform were discovered that might be related to our observations:

Facebook Developer Community Thread #228059956945288 Facebook Developer Community Thread #865649351807229 Facebook Support Bug #266015026245985

Proposed Solution:

Given the discrepancies observed and the insights from the customer's report in Issue #31787, it's crucial to re-evaluate our data retrieval method from the Facebook API. The customer suggests using the synchronous form of the request with the GET method on the same endpoint, instead of the POST and Ad Reports method. Check Facebook's Best Practices for more info.

Relevant log output

No response

Contribute

matheuzinoficial commented 9 months ago

Up! I noticed the same and also opened a similar issue days ago. This is actually making me lose some customers.

klsoper commented 8 months ago

@tolik0 what is the latest here?

tolik0 commented 8 months ago

@klsoper We'll wait until the end of this week to see if Facebook addresses this issue. If they don't, we'll begin working on it next week.

klsoper commented 8 months ago

thanks for the update @tolik0 !

pgrant87 commented 8 months ago

Hi all, just wanted to check on the status here, it doesn't look like any of the listed FB issues have been resolved yet

erica-airbyte commented 8 months ago

@tolik0 can you update here what direction we decided to go with this issue.

tolik0 commented 8 months ago

Update:

We have opened a new issue with Facebook, as the previous bug concerning the actions and action_values data discrepancies was closed without a resolution. To stay informed on the latest developments and updates regarding this matter, please follow the discussion in the Facebook Developer Community thread. We appreciate your patience and will keep the Airbyte community informed as we await a fix from Facebook.