pingcap-inc / tidb2dw

Replicate data changes from TiDB to Data Warehouses in real-time
MIT License
13 stars 7 forks source link

BigQuery: missing field #97

Closed eugen-korentsov closed 7 months ago

eugen-korentsov commented 7 months ago

Schema:

CREATE TABLE `testy_test_force_update_request` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `test_id` int(11) unsigned NOT NULL,
  `testy_test_id` int(11) unsigned NOT NULL,
  `is_affected` tinyint(3) unsigned NOT NULL DEFAULT '0',
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  UNIQUE KEY `sefur_unique_index` (`test_id`,`testy_test_id`),
  KEY `sefur_test_id_testy_test_id_index` (`test_id`,`testy_test_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=5930434

updated_at field isn't created in bigquery, in csv 6 columns, it result to BigQuery error during import:

Too many values in line. Found 6 column(s) when expecting 5.
wd0517 commented 7 months ago

This commit causes the bug.

mysql> SELECT COLUMN_NAME, COLUMN_DEFAULT, IS_NULLABLE, DATA_TYPE,  CHARACTER_MAXIMUM_LENGTH, NUMERIC_PRECISION, NUMERIC_SCALE, DATETIME_PRECISION, COLUMN_TYPE, EXTRA FROM information_schema.columns WHERE table_schema = 'test' and table_name = 'testy_test_force_update_request';
+---------------+-------------------+-------------+-----------+--------------------------+-------------------+---------------+--------------------+---------------------+-----------------------------------------------+
| COLUMN_NAME   | COLUMN_DEFAULT    | IS_NULLABLE | DATA_TYPE | CHARACTER_MAXIMUM_LENGTH | NUMERIC_PRECISION | NUMERIC_SCALE | DATETIME_PRECISION | COLUMN_TYPE         | EXTRA                                         |
+---------------+-------------------+-------------+-----------+--------------------------+-------------------+---------------+--------------------+---------------------+-----------------------------------------------+
| id            | NULL              | NO          | int       |                     NULL |                11 |             0 |               NULL | int(11) unsigned    | auto_increment                                |
| test_id       | NULL              | NO          | int       |                     NULL |                11 |             0 |               NULL | int(11) unsigned    |                                               |
| testy_test_id | NULL              | NO          | int       |                     NULL |                11 |             0 |               NULL | int(11) unsigned    |                                               |
| is_affected   | 0                 | NO          | tinyint   |                     NULL |                 3 |             0 |               NULL | tinyint(3) unsigned |                                               |
| created_at    | CURRENT_TIMESTAMP | NO          | timestamp |                     NULL |              NULL |          NULL |                  0 | timestamp           |                                               |
| updated_at    | CURRENT_TIMESTAMP | YES         | datetime  |                     NULL |              NULL |          NULL |                  0 | datetime            | DEFAULT_GENERATED on update CURRENT_TIMESTAMP |
+---------------+-------------------+-------------+-----------+--------------------------+-------------------+---------------+--------------------+---------------------+-----------------------------------------------+
6 rows in set (0.01 sec)

@Lloyd-Pottiger Maybe we should only exclude VIRTUAL GENERATED column?

Lloyd-Pottiger commented 7 months ago

@eugen-korentsov @wd0517 Thank you~

b892f265e6c4cb02397e4b0fd14d834a035b093d fix this issue~