Closed danieldiamond closed 2 years ago
Thanks @danieldiamond for reporting this. Sherif added it to the connector roadmap (estimated to 15-oct). Do you want to work on this issue? I can assign it to you.
@gaart
more specific context on this issue. loading the calls
and call_legs
tables into the destination. I only see exactly 1000 records in each table. looking further at the calls, i noticed created_at
dates are not necessarily recent, with some as far back as 2019
bringing private slack conversation here for visibility. the specific issue is pagination. apparently airbyte's test account does not have enough data to verify this. as mentioned above, i am seeing exactly 1k of calls
records coming through (we have more than that on our account). the issue here is to investigate why the sync loads 1k of records and moves on.
@gaart do you identify the issue? Can we publish the updated connector you in PR please?
@danieldiamond still working on it, quite complicated to prepare a testing dataset on this API, so it takes time
Is there any updates on this? From what i can tell the exact same issue was resolved for zendesk chat: https://github.com/airbytehq/airbyte/pull/7210 Can we not just use the same solution here?
Hi @danieldiamond , sorry for the late response. You are completely right, Zendesk Chat had the same fix already (and also was migrated to CDK). To guarantee a minimum level of quality the fix for Zendesk Talk required migration to CDK as well. The fix is ready, it awaits internal review and soon will be merged and published.
Enviroment
0.30.2-alpha
Current Behavior
Only receiving 1000 calls for each sync - the incremental append stream
Expected Behavior
I would expect all calls to be synced
Logs
LOG
``` 2021-10-01 00:01:34 INFO () WorkerRun(call):42 - Executing worker wrapper. Airbyte version: 0.30.2-alpha 2021-10-01 00:03:50 INFO () TemporalAttemptExecution(get):94 - Executing worker wrapper. Airbyte version: 0.30.2-alpha 2021-10-01 00:03:50 WARN () Databases(createPostgresDatabaseWithRetry):38 - Waiting for database to become available... 2021-10-01 00:03:50 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready... 2021-10-01 00:03:50 INFO () Databases(createPostgresDatabaseWithRetry):55 - Database available! 2021-10-01 00:03:50 INFO () DefaultReplicationWorker(run):82 - start sync worker. job id: 4077 attempt id: 0 2021-10-01 00:03:50 INFO () DefaultReplicationWorker(run):91 - configured sync modes: {null.current_queue_activity=full_refresh - append, null.ivrs=full_refresh - append, null.account_overview=full_refresh - append, null.greetings=full_refresh - append, null.call_legs=incremental - append, null.ivr_menus=full_refresh - append, null.agents_activity=full_refresh - append, null.addresses=full_refresh - append, null.agents_overview=full_refresh - append, null.greeting_categories=full_refresh - append, null.ivr_routes=full_refresh - append, null.calls=incremental - append, null.phone_numbers=full_refresh - append} 2021-10-01 00:03:50 INFO () DefaultAirbyteDestination(start):58 - Running destination... 2021-10-01 00:03:50 INFO () LineGobbler(voidCall):65 - Checking if airbyte/destination-snowflake:0.3.14 exists... 2021-10-01 00:03:51 INFO () LineGobbler(voidCall):65 - airbyte/destination-snowflake:0.3.14 was found locally. 2021-10-01 00:03:51 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4077/0 --network host --log-driver none airbyte/destination-snowflake:0.3.14 write --config destination_config.json --catalog destination_catalog.json 2021-10-01 00:03:51 INFO () LineGobbler(voidCall):65 - Checking if airbyte/source-zendesk-talk:0.1.2 exists... 2021-10-01 00:03:51 INFO () LineGobbler(voidCall):65 - airbyte/source-zendesk-talk:0.1.2 was found locally. 2021-10-01 00:03:51 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4077/0 --network host --log-driver none airbyte/source-zendesk-talk:0.1.2 read --config source_config.json --catalog source_catalog.json --state input_state.json 2021-10-01 00:03:51 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):226 - Destination output thread started. 2021-10-01 00:03:51 INFO () DefaultReplicationWorker(run):119 - Waiting for source thread to join. 2021-10-01 00:03:51 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):190 - Replication thread started. 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.d.s.SnowflakeDestination(main):81 - {} - starting destination: class io.airbyte.integrations.destination.snowflake.SnowflakeDestination 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.d.j.c.SwitchingDestination(getConsumer):83 - {} - Using destination type: COPY_S3 2021-10-01 00:03:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-01 00:03:55 [32mINFO[m i.a.i.d.j.c.s.S3StreamCopier(Steps to Reproduce
Are you willing to submit a PR?
Sure