Closed marcosmarxm closed 3 years ago
After removing both events
streams look the sync is processing.
Need to implement rate limit handling. An example may be found here.
Also IssueEvents
stream has wrong cursor_field
.
Any update on this?
Short term work-around has been to disable the events
and issue_events
streams.
If there is also a problem with rate-limiting, it would be good for that to be confirmed.
@Zirochkaa can you update the status? The PR is almost finished, as far I saw.
Hello :) Yes, it's almost done. There were few issues:
cursor_field
for IssueEvents
stream was incorrect;Issues
, Comments
and PullRequests
streams;All of the above issues are fixed now.
Right now I'm working on optimizing PullRequests
stream (the way how we read for the first time and subsequent times). Code should be on code review today.
Hi @garden-of-delete , We just released version 0.1.2. You can upgrade your connector from the Admin page. Please let us if the issue was fixed. Thanks
@keu Thanks so much! Will get into testing and report back tomorrow.
LOG
``` 2021-07-07 18:34:39 INFO () WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.27.0-alpha 2021-07-07 18:34:39 INFO () TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.27.0-alpha 2021-07-07 18:34:39 INFO () DefaultReplicationWorker(run):102 - start sync worker. job id: 415 attempt id: 0 2021-07-07 18:34:39 INFO () DefaultReplicationWorker(run):111 - configured sync modes: {null.commit_comments=full_refresh - append, null.pull_requests=full_refresh - append, null.comments=full_refresh - append, null.commits=full_refresh - append, null.issue_labels=full_refresh - append, null.assignees=full_refresh - append, null.issue_events=full_refresh - append, null.issues=full_refresh - append, null.teams=full_refresh - append, null.issue_milestones=full_refresh - append, null.reviews=full_refresh - append, null.releases=full_refresh - append, null.projects=full_refresh - append, null.events=full_refresh - append, null.collaborators=full_refresh - append, null.stargazers=full_refresh - append} 2021-07-07 18:34:39 INFO () DefaultAirbyteDestination(start):78 - Running destination... 2021-07-07 18:34:39 INFO () LineGobbler(voidCall):85 - Checking if airbyte/destination-bigquery:0.3.7 exists... 2021-07-07 18:34:39 INFO () LineGobbler(voidCall):85 - airbyte/destination-bigquery:0.3.7 was found locally. 2021-07-07 18:34:39 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/415/0 --network host --log-driver none airbyte/destination-bigquery:0.3.7 write --config destination_config.json --catalog destination_catalog.json 2021-07-07 18:34:39 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-github:0.1.1 exists... 2021-07-07 18:34:39 INFO () LineGobbler(voidCall):85 - airbyte/source-github:0.1.1 was found locally. 2021-07-07 18:34:39 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/415/0 --network host --log-driver none airbyte/source-github:0.1.1 read --config source_config.json --catalog source_catalog.json 2021-07-07 18:34:39 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):243 - Destination output thread started. 2021-07-07 18:34:39 INFO () DefaultReplicationWorker(run):139 - Waiting for source thread to join. 2021-07-07 18:34:39 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):207 - Replication thread started. 2021-07-07 18:34:40 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Starting syncing SourceGithub 2021-07-07 18:34:40 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: assignees 2021-07-07 18:34:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:41 [32mINFO[m i.a.i.d.b.BigQueryDestination(main):278 - {} - starting destination: class io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-07-07 18:34:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:41 [32mINFO[m i.a.i.b.IntegrationRunner(run):78 - {} - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-07-07 18:34:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:41 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2021-07-07 18:34:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:41 [32mINFO[m i.a.i.b.IntegrationRunner(run):82 - {} - Command: WRITE 2021-07-07 18:34:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:41 [32mINFO[m i.a.i.b.IntegrationRunner(run):83 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2021-07-07 18:34:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:43 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_oyo_github_superset_assignees}} created successfully 2021-07-07 18:34:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:43 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_kdt_github_superset_collaborators}} created successfully 2021-07-07 18:34:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:44 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_dcb_github_superset_comments}} created successfully 2021-07-07 18:34:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:44 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_jig_github_superset_commit_comments}} created successfully 2021-07-07 18:34:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:45 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_lca_github_superset_commits}} created successfully 2021-07-07 18:34:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:45 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_qdk_github_superset_events}} created successfully 2021-07-07 18:34:46 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:46 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_hff_github_superset_issue_events}} created successfully 2021-07-07 18:34:46 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:46 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_fzh_github_superset_issue_labels}} created successfully 2021-07-07 18:34:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:47 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_wqv_github_superset_issue_milestones}} created successfully 2021-07-07 18:34:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:47 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_fnl_github_superset_issues}} created successfully 2021-07-07 18:34:48 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:48 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_gha_github_superset_projects}} created successfully 2021-07-07 18:34:48 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:48 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_vnl_github_superset_pull_requests}} created successfully 2021-07-07 18:34:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:49 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_wpb_github_superset_releases}} created successfully 2021-07-07 18:34:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:49 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_csr_github_superset_reviews}} created successfully 2021-07-07 18:34:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:50 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_rbq_github_superset_stargazers}} created successfully 2021-07-07 18:34:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-07 18:34:50 [32mINFO[m i.a.i.d.b.BigQueryUtils(createTable):83 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=base, tableId=_airbyte_tmp_vac_github_superset_teams}} created successfully 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 61 records from assignees stream 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: collaborators 2021-07-07 18:36:47 WARN () DefaultAirbyteStreamFactory(internalLog):109 - 403 Client Error: Forbidden for url: https://api.github.com/repos/apache/superset/collaborators?per_page=100 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 0 records from collaborators stream 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: comments 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Caught retryable error after 1 tries. Waiting 5 seconds then retrying... 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Caught retryable error after 2 tries. Waiting 10 seconds then retrying... 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Caught retryable error after 3 tries. Waiting 20 seconds then retrying... 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Caught retryable error after 4 tries. Waiting 40 seconds then retrying... 2021-07-07 18:36:47 WARN () DefaultAirbyteStreamFactory(internalLog):109 - 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 0 records from comments stream 2021-07-07 18:36:47 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: commit_comments 2021-07-07 18:36:48 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 131 records from commit_comments stream 2021-07-07 18:36:48 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: commits 2021-07-07 18:36:51 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 1000 2021-07-07 18:36:55 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 2000 2021-07-07 18:37:00 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 3000 2021-07-07 18:37:05 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 4000 2021-07-07 18:37:08 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 5000 2021-07-07 18:37:12 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 6000 2021-07-07 18:37:15 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 7000 2021-07-07 18:37:18 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):220 - Records read: 8000 2021-07-07 18:37:20 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 7847 records from commits stream 2021-07-07 18:37:20 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: events 2021-07-07 18:37:23 ERROR () LineGobbler(voidCall):85 - Traceback (most recent call last): 2021-07-07 18:37:23 ERROR () LineGobbler(voidCall):85 - File "/airbyte/integration_code/main.py", line 33, inNot at the moment