search

ClickHouse / clickhouse-jdbc-bridge

A JDBC proxy from ClickHouse to external databases
Apache License 2.0
167 stars 60 forks source link

jdbc-bridge seems broken with latest CH release (21.5.5.12) #101

Closed wallflower closed 3 years ago

wallflower commented 3 years ago

I am not sure if this should be posted in this repo or the main clickhouse repository (I did post an issue there as well. I will close or the other depending what I hear)?

After upgrading my docker-compose file to the lastest version of clickhouse, the jdbc-bridge no longer works. Every query against it returns the same type of error:

SQL Error [86]: ClickHouse exception, code: 86, host: ***.***.com, port: 8123; Code: 86, e.displayText() = DB::Exception: Received error from remote server /?connection_string=&max_block_size=65505. HTTP status code: 500 Internal Server Error, body: At least one column is needed. (version 21.5.5.12 (official build))

If I downgrade clickhouse to 21.4.7.3 everything works again.

Here is the full stack trace in case it helps:

jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:288} - [/columns_info] Parameters:
jdbc-bridge_1  | connection_string:
jdbc-bridge_1  | table: show datasources
jdbc-bridge_1  | external_table_functions_use_nulls: true
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [INFO ] {JdbcBridgeVerticle:359} - Raw query:
jdbc-bridge_1  | show datasources
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {NamedDataSource:438} - Inferring columns: schema=[], query=[show datasources]
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:404} - Columns info:
jdbc-bridge_1  | [columns format version: 1
jdbc-bridge_1  | 10 columns:
jdbc-bridge_1  | `name` Nullable(String)
jdbc-bridge_1  | `is_alias` Nullable(UInt8)
jdbc-bridge_1  | `instance` Nullable(Int32)
jdbc-bridge_1  | `create_datetime` Nullable(DateTime)
jdbc-bridge_1  | `type` Nullable(String)
jdbc-bridge_1  | `parameters` Nullable(String)
jdbc-bridge_1  | `defaults` Nullable(String)
jdbc-bridge_1  | `custom_columns` Nullable(String)
jdbc-bridge_1  | `cache_usage` Nullable(String)
jdbc-bridge_1  | `pool_usage` Nullable(String)
jdbc-bridge_1  | ]
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:286} - [/ping] Context:
jdbc-bridge_1  | {__body-handled=true}
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:287} - [/ping] Headers:
jdbc-bridge_1  | Host: jdbc-bridge
jdbc-bridge_1  | Connection: Close
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:288} - [/ping] Parameters:
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:286} - [/ping] Context:
jdbc-bridge_1  | {__body-handled=true}
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:287} - [/ping] Headers:
jdbc-bridge_1  | Host: jdbc-bridge
jdbc-bridge_1  | Connection: Close
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:288} - [/ping] Parameters:
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:286} - [/identifier_quote] Context:
jdbc-bridge_1  | {__body-handled=true}
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:287} - [/identifier_quote] Headers:
jdbc-bridge_1  | Host: jdbc-bridge
jdbc-bridge_1  | Connection: Close
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:288} - [/identifier_quote] Parameters:
jdbc-bridge_1  | connection_string:
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:286} - [/] Context:
jdbc-bridge_1  | {__body-handled=true}
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:287} - [/] Headers:
jdbc-bridge_1  | Host: jdbc-bridge
jdbc-bridge_1  | Transfer-Encoding: chunked
jdbc-bridge_1  | Connection: Close
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [DEBUG] {JdbcBridgeVerticle:288} - [/] Parameters:
jdbc-bridge_1  | connection_string:
jdbc-bridge_1  | max_block_size: 65505
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-worker-thread-10] [DEBUG] {JdbcBridgeVerticle:439} - Generated query:
jdbc-bridge_1  |
jdbc-bridge_1  | Normalized query:
jdbc-bridge_1  |
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-worker-thread-10] [DEBUG] {ResponseWriter:45} - Start Time=1621891820242, Timeout=-1000, Max Block Size=65505
jdbc-bridge_1  | 2021-05-24 21:30:20.020 [vert.x-eventloop-thread-0] [ERROR] {JdbcBridgeVerticle:316} - Failed to respond
jdbc-bridge_1  | java.lang.IllegalArgumentException: At least one column is needed.
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.core.TableDefinition.<init>(TableDefinition.java:86)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.core.TableDefinition.fromString(TableDefinition.java:371)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.core.QueryParser.getTable(QueryParser.java:116)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle.handleQuery(JdbcBridgeVerticle.java:464)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle$$Lambda$113/0x00000000b81097e0.handle(Unknown Source)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$207/0x00000000b850bd90.handle(Unknown Source)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.vertx.core.impl.ContextImpl$$Lambda$93/0x00000000b8091c40.run(Unknown Source)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.vertx.core.impl.TaskQueue$$Lambda$76/0x00000000982df370.run(Unknown Source)
jdbc-bridge_1  |        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
jdbc-bridge_1  |        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
jdbc-bridge_1  |        at ru.yandex.clickhouse.jdbcbridge.internal.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
jdbc-bridge_1  |        at java.lang.Thread.run(Thread.java:823)

Here is my docker-compose file:

version: "2"

# Only 'ch-server' and 'jdbc-bridge' are mandatory.
# You may remove any db-xxx to save memory.
services:
  ch-server:
    image: yandex/clickhouse-server:21.5.5.12  #21.4.7.3
    ports:
       - 8123:8123
#       - 9000:9000
       - 9009:9009
    hostname: ch-server
    volumes:
      - ./ch-server/config:/etc/clickhouse-server/config.d
      #- ./ch-server/data:/var/lib/clickhouse
      #- ./ch-server/logs:/var/log/clickhouse
    mem_limit: 512m
    restart: always

  jdbc-bridge:
    image: yandex/clickhouse-jdbc-bridge:2.0.1
    hostname: jdbc-bridge
    # In general you don't need to define any environment variable
    # Below are all default settings just for demonstration
    environment:
      CONFIG_DIR: config # configuration directory
      HTTPD_CONFIG_FILE: httpd.json # httpd configuration file
      SERVER_CONFIG_FILE: server.json # server configuration file
      VERTX_CONFIG_FILE: vertx.json # vertx configuration file
      DATASOURCE_CONFIG_DIR: datasources # named datasource directory
      DRIVER_DIR: drivers # driver directory
      EXTENSION_DIR: extensions # extension directory
      QUERY_CONFIG_DIR: queries # named query directory
      CUSTOM_DRIVER_LOADER: "true" # whether use custom driver loader or not
      JDBC_BRIDGE_JVM_OPTS: # use CPU and memory allocated by container

    # You may want to keep datasources, queries, SQL scripts, and maybe drivers in a git repo
    volumes:
      - ./jdbc-bridge/config:/app/config
      - ./jdbc-bridge/drivers:/app/drivers
      - ./jdbc-bridge/scripts:/app/scripts
    mem_limit: 512m
    restart: always
zhicwu commented 3 years ago

Thanks for reporting the issue. From the log you posted, it seems there's parsing issue leading to empty query passed to target database. What was your query on clickhouse? It could also be an issue in jdbc bridge.

Mean time I'll add integration test in clickhouse to prevent this kind of issue from happening again.

wallflower commented 3 years ago

I was just doing, the following generic query:

select * from jdbc('', 'show datasources')

There are data sources, and they show up when I revert to 21.4.7.3

zhicwu commented 3 years ago

Thanks @wallflower. Does a query like select * from jdbc('ds', 'select * from table') or select * from jdbc('ds', 'table') work for you? Just trying to understand if all queries are impacted. I'll take a look at the issue today.

zhicwu commented 3 years ago

Link to Clickhouse/Clickhouse#24467

wallflower commented 3 years ago

The queries you provided, give a different and seemingly correct response, as I do not have a ds datasource.

Running:

select * from jdbc('ds', 'select * from table')

Results in:

SQL Error [86]: ClickHouse exception, code: 86, host: ***.***.com, port: 8123; Code: 86, e.displayText() = DB::Exception: Received error from remote server /columns_info?connection_string=ds&table=select%20%2A%20from%20table&external_table_functions_use_nulls=true. HTTP status code: 500 Internal Server Error, body: NamedDataSource [ds] does not exist! (version 21.5.5.12 (official build))
zhicwu commented 3 years ago

Thanks again. It's caused by server side changes and I'll try to fix it today.