search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
8.94k stars 1.79k forks source link

AirByte Connector Not Passing Checks when Creating a New Destination #28239

Closed brmei closed 1 year ago

brmei commented 1 year ago

Steps to reproduce the behavior (Required)

This tutorial was originally referenced: https://github.com/StarRocks/starrocks/discussions/23713

  1. Code from this repository is compiled and used instead of the StarRocks Airbyte connector: https://github.com/brmei/starrocks-airbyte-connector Image can be found here tagged as dev2: https://hub.docker.com/r/brianmei/destination-starrocks

  2. An instance of AirByte (version 0.50.11) is deployed by following these instructions: https://docs.airbyte.com/quickstart/deploy-airbyte/

  3. A cluster from CelerData Cloud is used. It is version 3.03-saas and running on AWS. Here is a screenshot of the overview.

Screenshot 2023-07-28 at 3 25 06 PM
  1. Create custom AirByte destination connector for StarRocks as shown in tutorial, using brianmei/destination-starrocks as the docker repository and dev2 as the tag
Screenshot 2023-07-28 at 4 31 01 PM
  1. Create destination using the following settings
Screenshot 2023-07-28 at 4 33 21 PM

Expected behavior (Required)

StarRocks AirByte connector passes tests

Real behavior (Required)

Configuration check fails with the message Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Here are the logs

2023-07-28 22:39:56 INFO i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/0339caef-5847-4363-9527-d65ce46ae6c9/0/logs.log
2023-07-28 22:39:56 INFO i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.50.11
2023-07-28 22:39:56 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):237 - Attempt 0 to save workflow id for cancellation
2023-07-28 22:39:56 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-07-28 22:39:56 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-07-28 22:39:56 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-07-28 22:39:56 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-07-28 22:39:56 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-07-28 22:39:56 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-07-28 22:39:56 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-07-28 22:39:56 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-07-28 22:39:56 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if brianmei/destination-starrocks:dev2 exists...
2023-07-28 22:39:57 INFO i.a.c.i.LineGobbler(voidCall):149 - brianmei/destination-starrocks:dev2 was found locally.
2023-07-28 22:39:57 INFO i.a.w.p.DockerProcessFactory(create):142 - Creating docker container = destination-starrocks-check-0339caef-5847-4363-9527-d65ce46ae6c9-0-fcmlb with resources io.airbyte.config.ResourceRequirements@9561bd0[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-07-28 22:39:57 INFO i.a.w.p.DockerProcessFactory(create):195 - Preparing command: docker run --rm --init -i -w /data/0339caef-5847-4363-9527-d65ce46ae6c9/0 --log-driver none --name destination-starrocks-check-0339caef-5847-4363-9527-d65ce46ae6c9-0-fcmlb --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=brianmei/destination-starrocks:dev2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.11 -e WORKER_JOB_ID=0339caef-5847-4363-9527-d65ce46ae6c9 brianmei/destination-starrocks:dev2 check --config source_config.json
2023-07-28 22:39:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):177 - Reading messages from protocol version 0.2.0
2023-07-28 22:39:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=source_config.json}
2023-07-28 22:39:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):106 Running integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination
2023-07-28 22:39:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):107 Command: CHECK
2023-07-28 22:39:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):108 Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-07-28 22:39:59 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-28 22:39:59 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-28 22:39:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):195 Completed integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination
2023-07-28 22:39:59 INFO i.a.w.g.DefaultCheckConnectionWorker(run):117 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@223e3364[status=failed,message=Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.,additionalProperties={}]
2023-07-28 22:39:59 INFO i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-07-28 22:39:59 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-07-28 22:39:59 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-07-28 22:39:59 INFO i.a.c.i.LineGobbler(voidCall):149 -

StarRocks version (Required)

3.0.3 fe5e3a1

brmei commented 1 year ago

Upgrading the CelerData cluster version to 3.1.0-rc01 solves this issue