search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.3k stars 4.15k forks source link

[source-clickhouse] Always tries to connect to `localhost` despite setting `host` in the configuration. #37498

Open tomlynchRNA opened 7 months ago

tomlynchRNA commented 7 months ago

Connector Name

source-clickhouse

Connector Version

0.22.2

What step the error happened?

Configuring a new connector

Relevant information

image

Relevant log output

Internal message: io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60001ms (total=0, active=0, idle=0, waiting=0)
Failure origin: source
Failure type: config_error

-----

    at java.base/java.net.Socket.connect(Socket.java:751)
    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
    at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:280)
    at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:386)
    at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:408)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1304)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1052)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1446)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1417)
    at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:226)
    at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:101)
    at com.clickhouse.client.AbstractClient.execute(AbstractClient.java:296)
    at com.clickhouse.client.ClickHouseClientBuilder$Agent.sendOnce(ClickHouseClientBuilder.java:191)
    at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:202)
    at com.clickhouse.client.ClickHouseClientBuilder$Agent.execute(ClickHouseClientBuilder.java:234)
    at com.clickhouse.client.ClickHouseRequest.execute(ClickHouseRequest.java:1748)
    at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:79)
    ... 14 more

------------

2024-04-23 05:04:53 platform > airbyte/source-clickhouse:0.2.2 was found locally.
2024-04-23 05:04:53 platform > Creating docker container = source-clickhouse-check-56ecda80-77d5-4592-8bf7-f5de28a2f4eb-0-wdacf with resources io.airbyte.config.ResourceRequirements@210c1c20[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-04-23 05:04:53 platform > Preparing command: docker run --rm --init -i -w /data/56ecda80-77d5-4592-8bf7-f5de28a2f4eb/0 --log-driver none --name source-clickhouse-check-56ecda80-77d5-4592-8bf7-f5de28a2f4eb-0-wdacf --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-clickhouse:0.2.2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.57.2 -e WORKER_JOB_ID=56ecda80-77d5-4592-8bf7-f5de28a2f4eb airbyte/source-clickhouse:0.2.2 check --config source_config.json
2024-04-23 05:04:53 platform > Reading messages from protocol version 0.2.0
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 INFO i.a.i.s.c.ClickHouseSource(main):133 - starting source: class io.airbyte.integrations.source.clickhouse.ClickHouseSource
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {check=null, config=source_config.json}
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 INFO i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 INFO i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: CHECK
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 INFO i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-04-23 05:04:54 platform > 2024-04-23 05:04:54 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-04-23 05:04:55 platform > 2024-04-23 05:04:55 INFO i.a.c.i.b.s.SshTunnel(getInstance):252 - Starting connection with method: NO_TUNNEL
2024-04-23 05:04:55 platform > 2024-04-23 05:04:55 INFO c.z.h.HikariDataSource(<init>):79 - HikariPool-1 - Starting...
2024-04-23 05:04:55 platform > 2024-04-23 05:04:55 INFO c.z.h.HikariDataSource(<init>):81 - HikariPool-1 - Start completed.
2024-04-23 05:05:55 platform > 2024-04-23 05:05:55 INFO c.z.h.HikariDataSource(close):349 - HikariPool-1 - Shutdown initiated...
2024-04-23 05:05:55 platform > 2024-04-23 05:05:55 INFO c.z.h.HikariDataSource(close):351 - HikariPool-1 - Shutdown completed.
2024-04-23 05:05:55 platform > 2024-04-23 05:05:55 INFO i.a.c.i.b.IntegrationRunner(runInternal):231 - Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-04-23 05:05:55 platform > 2024-04-23 05:05:55 INFO i.a.i.s.c.ClickHouseSource(main):135 - completed source: class io.airbyte.integrations.source.clickhouse.ClickHouseSource
2024-04-23 05:05:55 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@2bf6aeba[status=failed,message=State code: 08000; Error code: 210; Message: Connection refused, server ClickHouseNode [uri=http://localhost:8123/my_database],additionalProperties={}]
2024-04-23 05:05:55 platform > 
2024-04-23 05:05:55 platform > ----- END CHECK -----
2024-04-23 05:05:55 platform >

Contribute

marcosmarxm commented 7 months ago

Thanks for reporting the issue @tomlynchRNA Clickhouse is a community connector and it isn't in the current roadmap for improvements. If you want to contribute fixing the issue please reach me out in Slack so I can provide you instructions to make the contribution 🎖️

octavia-squidington-iii commented 1 month ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.