search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.53k stars 4k forks source link

StarRocks Connector Not Passing Checks when Creating a New Destination #28835

Closed brmei closed 1 year ago

brmei commented 1 year ago

Connector Name

destination-starrocks

Connector Version

0.0.1

What step the error happened?

Configuring a new connector

Revelant information

Issue also exists on StarRocks: https://github.com/StarRocks/starrocks/issues/28239

This tutorial was originally referenced: https://github.com/StarRocks/starrocks/discussions/23713

  1. Code from this repository is compiled and used instead of the StarRocks Airbyte connector: https://github.com/brmei/starrocks-airbyte-connector Image can be found here tagged as dev2: https://hub.docker.com/r/brianmei/destination-starrocks

  2. An instance of AirByte (version 0.50.11) is deployed by following these instructions: https://docs.airbyte.com/quickstart/deploy-airbyte/

  3. A cluster from CelerData Cloud is used. It is version 3.03-saas and running on AWS. Here is a screenshot of the overview.

Screenshot 2023-07-28 at 3 25 06 PM
  1. Create custom AirByte destination connector for StarRocks as shown in tutorial, using brianmei/destination-starrocks as the docker repository and dev2 as the tag
Screenshot 2023-07-28 at 4 31 01 PM
  1. Create destination using the following settings
Screenshot 2023-07-28 at 4 33 21 PM

The StarRocks version is 3.0.3 fe5e3a1

Relevant log output

2023-07-28 22:06:39 INFO i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/c30901e3-a8a5-4f3c-af83-e4504b214d1c/0/logs.log
2023-07-28 22:06:39 INFO i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.50.5
2023-07-28 22:06:39 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):229 - Attempt 0 to save workflow id for cancellation
2023-07-28 22:06:39 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-07-28 22:06:39 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-07-28 22:06:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-07-28 22:06:39 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-07-28 22:06:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-07-28 22:06:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-07-28 22:06:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-07-28 22:06:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-07-28 22:06:39 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if brianmei/destination-starrocks:dev2 exists...
2023-07-28 22:06:39 INFO i.a.c.i.LineGobbler(voidCall):149 - brianmei/destination-starrocks:dev2 was found locally.
2023-07-28 22:06:39 INFO i.a.w.p.DockerProcessFactory(create):139 - Creating docker container = destination-starrocks-check-c30901e3-a8a5-4f3c-af83-e4504b214d1c-0-axktf with resources io.airbyte.config.ResourceRequirements@41679b28[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-07-28 22:06:39 INFO i.a.w.p.DockerProcessFactory(create):192 - Preparing command: docker run --rm --init -i -w /data/c30901e3-a8a5-4f3c-af83-e4504b214d1c/0 --log-driver none --name destination-starrocks-check-c30901e3-a8a5-4f3c-af83-e4504b214d1c-0-axktf --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=brianmei/destination-starrocks:dev2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.5 -e WORKER_JOB_ID=c30901e3-a8a5-4f3c-af83-e4504b214d1c brianmei/destination-starrocks:dev2 check --config source_config.json
2023-07-28 22:06:39 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):177 - Reading messages from protocol version 0.2.0
2023-07-28 22:06:41 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=source_config.json}
2023-07-28 22:06:41 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):106 Running integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination
2023-07-28 22:06:41 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):107 Command: CHECK
2023-07-28 22:06:41 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):108 Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-07-28 22:06:42 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-28 22:06:42 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-28 22:06:43 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):195 Completed integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination
2023-07-28 22:06:43 INFO i.a.w.g.DefaultCheckConnectionWorker(run):115 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@54fa03b6[status=failed,message=Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.,additionalProperties={}]
2023-07-28 22:06:43 INFO i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-07-28 22:06:43 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-07-28 22:06:43 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-07-28 22:06:43 INFO i.a.c.i.LineGobbler(voidCall):149 -

Contribute

brmei commented 1 year ago

Upgrading the CelerData cluster version to 3.1.0-rc01 solves this issue