allwefantasy / spark-binlog

A library for querying Binlog with Apache Spark structure streaming, for Spark SQL , DataFrames and [MLSQL](https://www.mlsql.tech).
Apache License 2.0
154 stars 54 forks source link

Wrong time format problem #17

Closed zhengqiangtan closed 4 years ago

zhengqiangtan commented 4 years ago

Hi,I found that parsing binlog from MySQL into the delta table has a problem with the time format, as follows:

mysql origin data 1、建表格式
last_login datetime DEFAULT NULL COMMENT '最后一次登录时间', persist_code varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL, reset_password_code varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL, last_name varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL, created_at datetime DEFAULT NULL, updated_at datetime DEFAULT NULL, avatar varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL, disabled int(11) DEFAULT '0'

2、源表数据 image

query delta data like this:

+------+---------------------+-----------+----------+---------------------------------+-----------+----------+---------+------------------------------------------+-------------------+-----------------------+------------------------------------------------------------+-------------------+---------+-----------------------+-----------------------+------+--------+ |id |email |mobile |first_name|password |permissions|department|activated|activation_code |activated_at |last_login |persist_code |reset_password_code|last_name|created_at |updated_at |avatar|disabled| +------+---------------------+-----------+----------+---------------------------------+-----------+----------+---------+------------------------------------------+-------------------+-----------------------+------------------------------------------------------------+-------------------+---------+-----------------------+-----------------------+------+--------+ |339286|123456@qq.com |1222222225|��������� |xizxaS3p10gl2Tahp |null |null |0 |null |null |49787-02-21 08:53:200.0|null |null |null |48463-08-18 12:16:400.0|49787-02-21 08:53:200.0|null |0 |

sqoop import from mysql is ok -----------------------+---------------+----------------------+------------+------------------------+------------------------+---------+-----------+-------------+--+ last_login | persist_code | reset_password_code | last_name | created_at | updated_at | avatar | disabled | pt | -----------------------+---------------+----------------------+------------+------------------------+------------------------+---------+-----------+-------------+--+ | 2017-10-25 20:26:56.0 | NULL | NULL | NULL | 2016-06-29 10:46:49.0 | 2017-10-25 20:26:56.0 | NULL | 0 | 2019-12-18 | -----------------------+---------------+----------------------+------------+------------------------+------------------------+---------+-----------+-------------+--+

Furthermore, not all data is a matter of time format,just new add , Have you ever had a similar problem?

allwefantasy commented 4 years ago

Fixed in master branch. If there is any other problem, please file a new issue or reopen this issue.