airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.04k stars 4.11k forks source link

Low code CDK: SubStreamSlicer doesn't skip empty parent stream #19795

Closed jj-nilbodee closed 1 year ago

jj-nilbodee commented 1 year ago
## Environment - **Airbyte version**: "0.40.22" - **OS Version / Instance**: GKE - **Deployment**: Kubenetes - **Source Connector and version**: Zoom 0.1.0 - **Destination Connector and version**: BigQuery (denormalized typed struct) 1.2.7 - **Step where error happened**: Sync Job ## Current Behavior - Zoom connectors implemented SubStreamslicer which causes the issues when the parent stream returns empty result back the child stream then later injected `None` as a parameter as a result for an empty parent stream. ## Expected Behavior - Child stream should skip making a request when a parent stream is empty. ## Logs ![Screenshot 2022-11-24 at 4 43 56 pm](https://user-images.githubusercontent.com/16779128/203833737-8ee4dd14-e3ef-4b0e-b121-b749034f1992.png) Logs from Zoom call logs indicate that when the parent stream returns meeting = [] it then make a request to /v2/meetings/None next which causes error. ## Steps to Reproduce 1. try using Zoom connector (although I think it's general problem regarding SubStreamSlicer) 2. have one user with no meeting at all 3. try fetching meeting-related streams ## Are you willing to submit a PR? Sorry, I tried looking at the CDK but didn't 100% sure what stream slicer is doing
marcosmarxm commented 1 year ago

Thanks for reporting this @jhajajaas. @girarda are you aware of this?

jj-nilbodee commented 1 year ago

@marcosmarxm @girarda is it possible if you guys can prioritise this? I'm being blocked by this. Wish I could submit the PR but I couldn't

girarda commented 1 year ago

@jhajajaas what version of the CDK are you using? I would recommend upgrading to at least 0.4.1 because this should be fixed by https://github.com/airbytehq/airbyte/pull/18529

jj-nilbodee commented 1 year ago

Thanks for pointing that @girarda. I've opened a PR for that.