search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.3k stars 4.15k forks source link

[source-snapchat] source lags 3 days #34282

Open rytis-molecule opened 10 months ago

rytis-molecule commented 10 months ago

Connector Name

Snapchat Marketing

Connector Version

0.3.0

What step the error happened?

None

Relevant information

Snapchat extracts lags 3 days. Problem with date slicing. Check log time and end time in the relevant log output. This was since october. Could not find any solution. Tried to recreate the source and change start time but nothing worked. Also start_time is never the same day. Would be nice to get the same day data.

Thanks

job_16024_attempt_1_txt.txt

Relevant log output

2024-01-15 23:30:12 source > ads_stats_daily stream slices: [.... {'start_time': '2023-12-14', 'end_time': '2024-01-13', 'id': 'fc84890d-bee8-4ee6-88ca-bcd13335418d'}]

Contribute

marcosmarxm commented 10 months ago

The reason this happens is metric report has a lag of 48h. You can read more about here: https://github.com/airbytehq/airbyte/blob/ae343436797360305c8df1d2b228fefcf92b6bd9/airbyte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/source.py#L794-L801

rytis-molecule commented 10 months ago

The reason this happens is metric report has a lag of 48h. You can read more about here:

https://github.com/airbytehq/airbyte/blob/ae343436797360305c8df1d2b228fefcf92b6bd9/airbyte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/source.py#L794-L801

But maybe we Could remove that 2 day lag and get the partial numbers? Clients what to see What is in snap chat dashboards. Would it be possible and then have a window of 3 days to update the numbers once they finalise? @marcosmarxm

marcosmarxm commented 10 months ago

@rytis-molecule you can use the end_date parameter in the connector and update using the API to always match today or yesterday. Can you run some tests and check what is the results using the latest date and compare with the lag result?

Your suggestion is a good way, update the documentation and implement the window date to update previous entries. Do you want to work on this?

rytis-molecule commented 10 months ago

@marcosmarxm sure thing. How should I submit a changed code? just clone snapchat, change test and push?

rytis-molecule commented 10 months ago

@rytis-molecule you can use the end_date parameter in the connector and update using the API to always match today or yesterday. Can you run some tests and check what is the results using the latest date and compare with the lag result?

Your suggestion is a good way, update the documentation and implement the window date to update previous entries. Do you want to work on this?

using end date works.