search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.22k stars 3.92k forks source link

[source-snapchat] source lags 3 days #34282

Open rytis-molecule opened 7 months ago

rytis-molecule commented 7 months ago

Connector Name

Snapchat Marketing

Connector Version

0.3.0

What step the error happened?

None

Relevant information

Snapchat extracts lags 3 days. Problem with date slicing. Check log time and end time in the relevant log output. This was since october. Could not find any solution. Tried to recreate the source and change start time but nothing worked. Also start_time is never the same day. Would be nice to get the same day data.

Thanks

job_16024_attempt_1_txt.txt

Relevant log output

2024-01-15 23:30:12 source > ads_stats_daily stream slices: [.... {'start_time': '2023-12-14', 'end_time': '2024-01-13', 'id': 'fc84890d-bee8-4ee6-88ca-bcd13335418d'}]

Contribute

marcosmarxm commented 7 months ago

The reason this happens is metric report has a lag of 48h. You can read more about here: https://github.com/airbytehq/airbyte/blob/ae343436797360305c8df1d2b228fefcf92b6bd9/airbyte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/source.py#L794-L801

rytis-molecule commented 7 months ago

The reason this happens is metric report has a lag of 48h. You can read more about here:

https://github.com/airbytehq/airbyte/blob/ae343436797360305c8df1d2b228fefcf92b6bd9/airbyte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/source.py#L794-L801

But maybe we Could remove that 2 day lag and get the partial numbers? Clients what to see What is in snap chat dashboards. Would it be possible and then have a window of 3 days to update the numbers once they finalise? @marcosmarxm

marcosmarxm commented 7 months ago

@rytis-molecule you can use the end_date parameter in the connector and update using the API to always match today or yesterday. Can you run some tests and check what is the results using the latest date and compare with the lag result?

Your suggestion is a good way, update the documentation and implement the window date to update previous entries. Do you want to work on this?

rytis-molecule commented 7 months ago

@marcosmarxm sure thing. How should I submit a changed code? just clone snapchat, change test and push?

rytis-molecule commented 7 months ago

@rytis-molecule you can use the end_date parameter in the connector and update using the API to always match today or yesterday. Can you run some tests and check what is the results using the latest date and compare with the lag result?

Your suggestion is a good way, update the documentation and implement the window date to update previous entries. Do you want to work on this?

using end date works.