search

pingcap-inc / tidb2dw

Replicate data changes from TiDB to Data Warehouses in real-time
MIT License
13 stars 7 forks source link

[enhancement] support more TiDB data types #44

Open zhangjinpeng87 opened 1 year ago

zhangjinpeng87 commented 1 year ago

Because snowflake doesn't support all data types in MySQL/TiDB database, we need a way to automatically transfer these unsupported data types from the upstream into some similar types downstream.

Snowflake unsupported data types:

eugen-korentsov commented 9 months ago

Same issue with bigquery.

eugen-korentsov commented 9 months ago 
[2024/01/17 09:21:44.427 +00:00] [ERROR] [snapshot.go:176] ["Failed to load snapshot"] [table=database.source_estates] [error="Unsupported TiDB type enum"] [errorVerbose="Unsupported TiDB type enum\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.GetBigQueryColumnTypeString\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/types.go:47\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.GetBigQueryColumnString\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/ddl.go:128\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.GenCreateSchema\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/sql.go:74\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.(*BigQueryConnector).CopyTableSchema\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/connector.go:99\ngithub.com/pingcap-inc/tidb2dw/replicate.(*SnapshotReplicateSession).Run\n\t/home/eugene/work/database/tidb2dw/replicate/snapshot.go:88\ngithub.com/pingcap-inc/tidb2dw/replicate.StartReplicateSnapshot\n\t/home/eugene/work/database/tidb2dw/replicate/snapshot.go:175\ngithub.com/pingcap-inc/tidb2dw/cmd.Replicate.func1\n\t/home/eugene/work/database/tidb2dw/cmd/core.go:271\nruntime.goexit\n\t/usr/lib/go-1.21/src/runtime/asm_amd64.s:1650\ngithub.com/pingcap/errors.AddStack\n\t/home/eugene/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/home/eugene/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/juju_adaptor.go:15\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.GetBigQueryColumnString\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/ddl.go:130\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.GenCreateSchema\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/sql.go:74\ngithub.com/pingcap-inc/tidb2dw/pkg/bigquerysql.(*BigQueryConnector).CopyTableSchema\n\t/home/eugene/work/database/tidb2dw/pkg/bigquerysql/connector.go:99\ngithub.com/pingcap-inc/tidb2dw/replicate.(*SnapshotReplicateSession).Run\n\t/home/eugene/work/database/tidb2dw/replicate/snapshot.go:88\ngithub.com/pingcap-inc/tidb2dw/replicate.StartReplicateSnapshot\n\t/home/eugene/work/database/tidb2dw/replicate/snapshot.go:175\ngithub.com/pingcap-inc/tidb2dw/cmd.Replicate.func1\n\t/home/eugene/work/database/tidb2dw/cmd/core.go:271\nruntime.goexit\n\t/usr/lib/go-1.21/src/runtime/asm_amd64.s:1650"]