search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.88k stars 3.82k forks source link

Source Postgres: Airbyte running locally cannot connect to locally running Google Cloud SQL Proxy #24751

Open ejstembler opened 1 year ago

ejstembler commented 1 year ago

Environment

Current Behavior

I’m running Airbyte locally on macOS (13.2.1 (22D68)) on localhost:8000. In the web UI I’m attempting to create a Postgres connection. I have Google Cloud SQL proxy running locally on localhost:5432. However, testing the connection fails:

The connection tests failed.

State code: 08001; Message: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

I’ve tried specifying 127.0.0.1 and 0.0.0.0 instead of localhost, I’ve also tried running Google Cloud SQL proxy on a different port than 5432. Still doesn’t work.

Incidentally, I can successfully connect to Google Cloud SQL Proxy via other means: DBVisualizer, JetBrains DataGrip, Ruby scripts, Python Script, etc…

Expected Behavior

Expected successful connection to local Google Cloud SQL Proxy.

Logs

2023-03-31 16:54:41 INFO i.a.w.t.TemporalAttemptExecution(get):142 - Docker volume job log path: /tmp/workspace/acd0e6d9-c234-446b-81c1-1715d29fa64a/0/logs.log
2023-03-31 16:54:41 INFO i.a.w.t.TemporalAttemptExecution(get):147 - Executing worker wrapper. Airbyte version: 0.42.0
2023-03-31 16:54:41 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation
2023-03-31 16:54:41 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-03-31 16:54:41 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-03-31 16:54:41 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-03-31 16:54:41 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-03-31 16:54:41 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-03-31 16:54:41 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-03-31 16:54:41 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-03-31 16:54:41 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-03-31 16:54:41 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-03-31 16:54:41 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-postgres:2.0.0 exists...
2023-03-31 16:54:41 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-postgres:2.0.0 was found locally.
2023-03-31 16:54:41 INFO i.a.w.p.DockerProcessFactory(create):130 - Creating docker container = source-postgres-check-acd0e6d9-c234-446b-81c1-1715d29fa64a-0-bhwfc with resources io.airbyte.config.ResourceRequirements@13aaee32[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null
2023-03-31 16:54:41 INFO i.a.w.p.DockerProcessFactory(create):175 - Preparing command: docker run --rm --init -i -w /data/acd0e6d9-c234-446b-81c1-1715d29fa64a/0 --log-driver none --name source-postgres-check-acd0e6d9-c234-446b-81c1-1715d29fa64a-0-bhwfc --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e STRICT_COMPARISON_NORMALIZATION_WORKSPACES= -e WORKER_CONNECTOR_IMAGE=airbyte/source-postgres:2.0.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e STRICT_COMPARISON_NORMALIZATION_TAG=strict_comparison2 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e FEATURE_FLAG_CLIENT= -e AIRBYTE_VERSION=0.42.0 -e WORKER_JOB_ID=acd0e6d9-c234-446b-81c1-1715d29fa64a airbyte/source-postgres:2.0.0 check --config source_config.json
2023-03-31 16:54:41 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):107 - Reading messages from protocol version 0.2.0
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.a.AdaptiveSourceRunner$Runner(getSource):73 Running source under deployment mode: OSS
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.a.AdaptiveSourceRunner$Runner(run):85 Starting source: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=source_config.json}
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.IntegrationRunner(runInternal):105 Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.IntegrationRunner(runInternal):106 Command: CHECK
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.IntegrationRunner(runInternal):107 Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-03-31 16:54:42 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):165 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-03-31 16:54:42 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):165 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: NO_TUNNEL
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.s.p.PostgresUtils(isCdc):53 using CDC: false
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting...
2023-03-31 16:54:42 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed.
2023-03-31 16:54:52 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated...
2023-03-31 16:54:52 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed.
2023-03-31 16:54:52 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.IntegrationRunner(runInternal):182 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-03-31 16:54:52 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):168 - INFO i.a.i.b.a.AdaptiveSourceRunner$Runner(run):87 Completed source: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-03-31 16:54:52 INFO i.a.w.g.DefaultCheckConnectionWorker(run):120 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@40d7aa39[status=failed,message=State code: 08001; Message: Connection to 0.0.0.0:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.]
2023-03-31 16:54:52 INFO i.a.w.t.TemporalAttemptExecution(get):169 - Stopping cancellation check scheduling...
2023-03-31 16:54:52 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-03-31 16:54:52 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-03-31 16:54:52 INFO i.a.c.i.LineGobbler(voidCall):149 -

Steps to Reproduce

  1. Run Google's cloud-sql-proxy local. Verify connection is working.

From Local Deployment:

  1. Clone Airbyte from GitHub
git clone https://github.com/airbytehq/airbyte.git
  1. Switch into Airbyte directory
cd airbyte
  1. Start Airbyte
./run-ab-platform.sh
  1. In your browser, just visit http://localhost:8000/
  2. You will be asked for a username and password. By default, that's username airbyte and password password. Once you deploy Airbyte to your servers, be sure to change these.
  3. Click the "Create your first connection" button and, select Postgres as the source type, and fill out the local cloud sql proxy connection information.
  4. Click the "Setup source" button.

References

ejstembler commented 1 year ago

I was able to get it to work after using the local machine's I.P. address. Both in cloud-sql-proxy via the --address flag, and by using the I.P. has the host in the web ui. Thanks to Sean Zicari on the Slack channel for that suggestion!

In any case, most people will not hard-code their machine's I.P. when using cloud-sql-proxy. The default is to use localhost. Perhaps this issue should remain open to better support the defaults which would be localhost, or 0.0.0.0, or 127.0.0.1?

octavia-squidington-iii commented 1 month ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.