search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
299 stars 16 forks source link

MongoDB - Empty ISODate column in Postgres destination #281

Closed alexismanuel closed 1 month ago

alexismanuel commented 2 months ago

Issue Description

source: SRC_STOREBOX_MONGO_DEV
target: DST_DWH_POSTGRES_STG

streams:
  test.test_coll:
    object: source_storebox_test.test_coll
    mode: full-refresh
  ...
•100% ➜ sling run -r test.yml -d                                                                            
2024-04-29 12:25:56 INF Sling Replication [1 streams] | SRC_STOREBOX_MONGO_DEV -> DST_DWH_POSTGRES_STG

2024-04-29 12:25:56 INF [1 / 1] running stream test.test_coll
2024-04-29 12:25:56 DBG Sling version: 1.2.9 (darwin arm64)
2024-04-29 12:25:56 DBG type is db-db
2024-04-29 12:25:56 DBG using source options: {"empty_as_null":false,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-04-29 12:25:56 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"}
2024-04-29 12:25:56 INF connecting to source database (mongodb)
2024-04-29 12:25:56 INF connecting to target database (postgres)
2024-04-29 12:25:56 DBG opened "postgres" connection (conn-postgres-DLo)
2024-04-29 12:25:56 INF reading from source database
2024-04-29 12:25:56 DBG {"collection":"test_coll","database":"test","filter":[],"options":{"limit":0,"projection":null}}
2024-04-29 12:25:56 INF writing to target database [mode: full-refresh]
2024-04-29 12:25:56 DBG drop table if exists "source_storebox_test"."test_coll_tmp"
2024-04-29 12:25:56 DBG table "source_storebox_test"."test_coll_tmp" dropped
2024-04-29 12:25:56 DBG create table if not exists "source_storebox_test"."test_coll_tmp" ("_id" varchar(65500),
"created_at" varchar(65500),
"id" integer)
2024-04-29 12:25:56 INF streaming data
2024-04-29 12:25:56 DBG select count(*) cnt from "source_storebox_test"."test_coll_tmp"
2024-04-29 12:25:56 DBG comparing checksums ["_id [text | character varying]","created_at [text | character varying]","id [integer | integer]"]
2024-04-29 12:25:56 DBG select sum(length("_id"::text)) as "_id", sum(length("created_at"::text)) as "created_at", sum(abs("id")) as "id" from "source_storebox_test"."test_coll_tmp"
2024-04-29 12:25:56 DBG drop table if exists "source_storebox_test"."test_coll"
2024-04-29 12:25:56 DBG table "source_storebox_test"."test_coll" dropped
2024-04-29 12:25:56 DBG create table if not exists "source_storebox_test"."test_coll" ("_id" varchar(65500),
"created_at" varchar(65500),
"id" integer)
2024-04-29 12:25:56 INF created table "source_storebox_test"."test_coll"
2024-04-29 12:25:56 DBG insert into "source_storebox_test"."test_coll" ("_id", "created_at", "id") select "_id", "created_at", "id" from "source_storebox_test"."test_coll_tmp"
2024-04-29 12:25:56 DBG inserted rows into "source_storebox_test"."test_coll" from temp table "source_storebox_test"."test_coll_tmp"
2024-04-29 12:25:56 INF inserted 1 rows into "source_storebox_test"."test_coll" in 0 secs [1 r/s] [28 B]
2024-04-29 12:25:56 DBG drop table if exists "source_storebox_test"."test_coll_tmp"
2024-04-29 12:25:56 DBG table "source_storebox_test"."test_coll_tmp" dropped
2024-04-29 12:25:56 DBG closed "postgres" connection (conn-postgres-DLo)
2024-04-29 12:25:56 INF execution succeeded

2024-04-29 12:25:57 INF Sling Replication Completed in 1s | SRC_STOREBOX_MONGO_DEV -> DST_DWH_POSTGRES_STG | 1 Successes | 0 Failures
flarco commented 2 months ago

Thanks for reporting, will take a look

flarco commented 1 month ago

Should be good in upcoming release https://github.com/slingdata-io/sling-cli/pull/289. Closing