airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.83k stars 4.06k forks source link

JDBC with 127.0.0.1 or localhost will not work but JDBC with non-127.0.0.1 works. using starrocks allin1 container with Airbyte connector test. #29558

Open alberttwong opened 1 year ago

alberttwong commented 1 year ago

Topic

networking issue with connector development

Revelant information

With 127.0.0.1 I get Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

But with 192.168.3.115 I get success.

192.168.3.115 is my IP for my machine.

{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):37 dbURL: jdbc:mysql://127.0.0.1:9030/demo?rewriteBatchedStatements=true&useUnicode=true&characterEncoding=utf8&autoReconnect=true"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):52 username: root, pwd: "}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):60 dbURL-withoutDB: jdbc:mysql://127.0.0.1:9030/"}}
{"type":"CONNECTION_STATUS","connectionStatus":{"status":"FAILED","message":"Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):226 Completed integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination"}}
atwong@Alberts-MacBook-Pro-3 airbyte % vi secrets/config.json
atwong@Alberts-MacBook-Pro-3 airbyte % docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-starrocks:dev check --config /secrets/config.json --catalog /sample_files/configured_catalog.json
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=/secrets/config.json}"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):130 Running integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):131 Command: CHECK"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):132 Integration config: IntegrationConfig{command=CHECK, configPath='/secrets/config.json', catalogPath='null', statePath='null'}"}}
{"type":"LOG","log":{"level":"WARN","message":"WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword"}}
{"type":"LOG","log":{"level":"WARN","message":"WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):37 dbURL: jdbc:mysql://192.168.3.115:9030/demo?rewriteBatchedStatements=true&useUnicode=true&characterEncoding=utf8&autoReconnect=true"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):52 username: root, pwd: "}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):60 dbURL-withoutDB: jdbc:mysql://192.168.3.115:9030/"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):65 query: create database if not exists demo;"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.StarRocksDestination(check):80 HTTP/HTTPS test connectivity: http://192.168.3.115:8040"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.StarRocksDestination(check):87 HTTP Response Code: 404"}}
{"type":"CONNECTION_STATUS","connectionStatus":{"status":"SUCCEEDED"}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):226 Completed integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination"}}

localhost has the same issue.

{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.d.s.SqlUtil(createJDBCConnection):60 dbURL-withoutDB: jdbc:mysql://localhost:9030/"}}
{"type":"CONNECTION_STATUS","connectionStatus":{"status":"FAILED","message":"Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}}
{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):226 Completed integration: io.airbyte.integrations.destination.starrocks.StarRocksDestination"}}

DBeaver works and I imagine it has to be JDBC but I don't understand how. Using the mysql 8.0.29 driver and using allin1 container.

alberttwong commented 1 year ago

https://github.com/StarRocks/starrocks/issues/29285

arthurbarros commented 1 month ago

@alberttwong any fix on this?

alberttwong commented 1 month ago

Unfortunately I'm not with StarRocks anymore. You'll have to reach out to them on their slack.