ClickHouse / clickhouse-connect

Python driver/sqlalchemy/superset connectors
Apache License 2.0
330 stars 64 forks source link

Unable to insert epoch timestamp using clickhouse connect 0.8.5 if timestamp format is DateTime64 with precision 6 #418

Closed wiseliu-finaccel closed 3 weeks ago

wiseliu-finaccel commented 3 weeks ago

Describe the bug

Failed to insert CSV file if the file contains 16 digit epoch to column with DateTime64(6) using clickhouse-connect python driver

Steps to reproduce

  1. use insert_file from import clickhouse_connect.driver.tools.insert_file
  2. insert CSV file that contains 16 digit epoch as DateTime64 precision 6

Expected behaviour

CSV file inserted successfully

Code example

import clickhouse_connect
from clickhouse_connect.driver.tools import insert_file

client = clickhouse_connect.get_client(host=server_ip, port=8123, username=username, password=clickhouse_password, database=database, session_id=session_id_str, compress=False)

insert_file(
            client=client,
            table='table',
            database='database',
            file_path=filename,
            fmt='CSV',
            column_names=['timestamp','instanceName','username','userHost','connectionid','queryid','operation','database','object','retcode'],
            settings={
                'format_csv_allow_single_quotes': True,
                'format_csv_delimiter': '~',
                'input_format_allow_errors_ratio': 0.1,
                'input_format_allow_errors_num': 100,
                'async_insert': 1, 
                'wait_for_async_insert': 1,
                'async_insert_max_data_size': 1000000,
                'async_insert_deduplicate': 1
            },
        )

clickhouse-connect and/or ClickHouse server logs

2024.11.01 12:31:32.746810 [ 3131966 ] {07d51f69-07ef-4c51-92b3-7389db865c14} <Error> executeQuery: Code: 41. DB::Exception: Cannot read DateTime: unexpected number of decimal digits: 16: (Already have 101 errors out of 101 rows, which is 1 of all rows): (at row 101)

Configuration

Environment

ClickHouse server

( timestamp DateTime64(6, 'Asia/Jakarta') CODEC(DoubleDelta, ZSTD), instanceName LowCardinality(String) CODEC(ZSTD), username LowCardinality(String) CODEC(ZSTD), host String DEFAULT '' CODEC(ZSTD), connectionid UInt64 CODEC(T64, ZSTD), queryid UInt64 CODEC(T64, ZSTD), operation LowCardinality(String) CODEC(ZSTD), database LowCardinality(String) CODEC(ZSTD), object String CODEC(ZSTD), retcode Int8 CODEC(T64, ZSTD), )


* Sample data for these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
genzgd commented 3 weeks ago

clickhouse-connect just passes the file directly to ClickHouse, which does all parsing. You should check the main ClickHouse repository for similar issues, and open a new one there if it hasn't already been filed (I vaguely recall something similar). You can create a reproducible example for a new issue using the official ClickHouse CLI client.