broken when try import the data from mysql to clickhouse

[justlooks]

hi, I try use clickhouse-mysql-data-reader to set the mysql - click house replication, but when I start click-mysql it fail to import data ,the row 5 data cause click-mysql broken ,I do not know why?

2019-01-16 15:03:01,284/1547622181.284754:DEBUG:Query: CREATE TABLE IF NOT EXISTS `xxx`.`xxx` (
    `primary_date_field` Date default today(),
    `id` Int32,
    `creator` Nullable(Int32),
    `modifier` Nullable(Int32),
    `gmt_create` DateTime,
    `pay_time` Nullable(DateTime),
    `gmt_modified` DateTime,
    `is_deleted` Nullable(String),
    `shop_id` Int32,
    `order_type` Int32,
    `parent_id` Nullable(Int32),
    `order_sn` String,
    `customer_id` Nullable(Int32),
    `customer_car_id` Int32,
    `expected_time` Nullable(DateTime),
    `goods_amount` Nullable(String),
    `service_amount` Nullable(String),
    `tax_amount` Nullable(String),
    `total_amount` Nullable(String),
    `postscript` Nullable(String),
    `order_status` String,
    `car_license` String,
    `car_brand_id` Nullable(Int32),
    `car_series_id` Nullable(Int32),
    `car_power_id` Nullable(Int32),
    `car_year_id` Nullable(Int32),
    `car_models_id` Nullable(Int32),
    `car_brand` Nullable(String),
    `car_series` Nullable(String),
    `car_power` Nullable(String),
    `car_year` Nullable(String),
    `car_models` Nullable(String),
    `car_company` Nullable(String),
    `import_info` Nullable(String),
    `customer_name` Nullable(String),
    `customer_mobile` Nullable(String),
    `vin` Nullable(String),
    `engine_no` Nullable(String),
    `receiver` Int32,
    `invoice_type` Nullable(Int32),
    `discount` Nullable(String),
    `receiver_name` Nullable(String),
    `operator_name` Nullable(String),
    `order_amount` Nullable(String),
    `pay_status` UInt8,
    `finish_time` Nullable(DateTime),
    `goods_count` Nullable(Int32),
    `service_count` Nullable(Int32),
    `car_alias` Nullable(String),
    `pre_discount_rate` Nullable(String),
    `pre_tax_amount` Nullable(String),
    `pre_preferentia_amount` Nullable(String),
    `pre_coupon_amount` Nullable(String),
    `pre_total_amount` Nullable(String),
    `pay_amount` Nullable(String),
    `sign_amount` Nullable(String),
    `other_insurance_company_id` Nullable(Int32),
    `other_insurance_company_name` Nullable(String),
    `goods_discount` Nullable(String),
    `service_discount` Nullable(String),
    `fee_amount` Nullable(String),
    `fee_discount` Nullable(String),
    `contact_name` Nullable(String),
    `contact_mobile` Nullable(String),
    `insurance_company_id` Nullable(Int32),
    `insurance_company_name` Nullable(String),
    `mileage` Nullable(String),
    `car_color` Nullable(String),
    `buy_time` Nullable(DateTime),
    `customer_address` Nullable(String),
    `oil_meter` Nullable(String),
    `is_notice` Nullable(Int8),
    `is_visit` Nullable(Int32),
    `is_lock` Nullable(String),
    `img_url` Nullable(String),
    `order_tag` UInt8,
    `ver` Nullable(String),
    `refer` UInt8,
    `upkeep_mileage` Nullable(String),
    `create_time` DateTime,
    `proxy_type` Int8,
    `channel_id` Int32,
    `channel_name` String,
    `order_discount_amount` Nullable(String),
    `confirm_time` DateTime,
    `expect_time` Nullable(DateTime),
    `down_payment` String,
    `company` Nullable(String)
) ENGINE = MergeTree(primary_date_field, (id,gmt_create,gmt_modified,order_type,order_sn,customer_car_id,order_status,car_license,receiver,pay_status,confirm_time), 8192)

......

Row 5:
Column 0,   name: buy_time,                     type: Nullable(DateTime), parsed text: ",,??,32,,,?A4369G,C"
ERROR: garbage after Nullable(DateTime): "addy [??],"

tail: write error: Broken pipe
2019-01-16 15:03:01,895/1547622181.895748:DEBUG:class:<class 'clickhouse_mysql.writer.processwriter.ProcessWriter'> process() done

I checked the csv file ,the row 5 is the following text

47352 1 1 2018-05-10 17:59:07 2018-05-10 17:59:07 2018-05-10 17:59:07 N 1 0 0 C000011805100001 411715 3455 20 \N 12.00 100.00 0.00 112.00 测试11 DDYFK 苏A4369G 32 446 0 0 84597 大众 (进口) Caddy [开迪] Caddy [开迪] \N wang 1G1BL52P7TR115520 1 0 0.00 tqmall tqmall 112.00 1 \N 1 1 1.00000000 0.00 0.00 0.00 112.00 112.00 112.00 0 \N 0.00 0.00 0.00 0.00 wang 0 \N \N \N 0 0 N \N 3 \N 0 2018-05-10 17:57:00 0 0 0.00 2018-05-10 17:59:07 \N 0.00

[alex-zaitsev]

There is probably a problem with non-ascii symbols. @sunsingerus , take a look

[justlooks]

I reinstall it with git master version ,now different error

# cat click-mysql2.sh
clickhouse-mysql \
    --src-host=10.100.3.4 \
    --src-user=clickhouse \
    --src-password=123456 \
    --migrate-table \
    --src-tables=test.legend_order_info \
    --dst-host=127.0.0.1

2019-01-18 15:11:42,512/1547795502.512598:CRITICAL:'TableMigrator' object has no attribute 'host'
=============
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/clickhouse_mysql/writer/chwriter.py", line 82, in insert
    self.client.execute(sql, rows)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/client.py", line 202, in execute
    query_id=query_id, types_check=types_check
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/client.py", line 351, in process_insert_query
    self.send_data(sample_block, data, types_check=types_check)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/client.py", line 375, in send_data
    self.connection.send_data(block)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/connection.py", line 476, in send_data
    self.block_out.write(block)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/streams/native.py", line 41, in write
    self.fout, types_check=block.types_check)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/columns/service.py", line 84, in write_column
    column.write_data(items, buf)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/columns/base.py", line 79, in write_data
    self._write_data(items, buf)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/columns/base.py", line 83, in _write_data
    self.write_items(prepared, buf)
  File "/usr/lib/python3.4/site-packages/clickhouse_driver/columns/stringcolumn.py", line 27, in write_items
    value = utf_8_encode(value)[0]
TypeError: Can't convert 'decimal.Decimal' object to str implicitly

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/clickhouse_mysql/tablemigrator.py", line 194, in migrate_one_table_data
    self.chwriter.insert(event)
  File "/usr/lib/python3.4/site-packages/clickhouse_mysql/writer/chwriter.py", line 88, in insert
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/clickhouse_mysql/main.py", line 140, in run
    migrator.migrate_all_tables_data()
  File "/usr/lib/python3.4/site-packages/clickhouse_mysql/tablemigrator.py", line 160, in migrate_all_tables_data
    self.migrate_one_table_data(db=db, table=table)
  File "/usr/lib/python3.4/site-packages/clickhouse_mysql/tablemigrator.py", line 200, in migrate_one_table_data
    self.host,
AttributeError: 'TableMigrator' object has no attribute 'host'
=============
'TableMigrator' object has no attribute 'host'

[justlooks]

I know why ,clickhouse-mysql convert mysql decimal type to String automatically ,but when migrate table data ,it can not convert decimal to string ,I think it is a bug , I bypass the problem by correcting all my decimal column manually.