search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
398 stars 27 forks source link

replication gets stuck #222

Closed alberttwong closed 6 months ago

alberttwong commented 6 months ago

Using https://atwong.medium.com/easiest-way-to-load-tpc-ds-data-into-postgresql-1ebd83871a07 to generate TPC DS data in Duckdb. 

atwong@Albert-CelerData ~ % duckdb duckdb.db
v0.10.0 20b1486d11
Enter ".help" for usage hints.
D INSTALL tpcds;
D LOAD tpcds;
D SELECT * FROM dsdgen(sf=1);
100% ▕████████████████████████████████████████████████████████████▏
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ 0 rows  │
└─────────┘
D

this generate 1 GB of TPC DS data.

atwong@Albert-CelerData ~ % sling run -d -r ./replication.yaml
2024-03-12 09:46:36 DBG processing wildcards for DUCKDB
2024-03-12 09:46:36 DBG getting tables for main.*
2024-03-12 09:46:36 INF Sling Replication [24 streams] | DUCKDB -> STARROCKSLOCAL

2024-03-12 09:46:36 INF [1 / 24] running stream "main"."call_center"
2024-03-12 09:46:36 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:46:36 DBG type is db-db
2024-03-12 09:46:36 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:46:36 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:46:36 INF connecting to source database (duckdb)
2024-03-12 09:46:36 INF connecting to target database (starrocks)
2024-03-12 09:46:36 INF reading from source database
2024-03-12 09:46:36 DBG select * from "main"."call_center"
2024-03-12 09:46:36 INF writing to target database [mode: full-refresh]
2024-03-12 09:46:36 DBG drop table if exists `main`.`call_center_tmp`
2024-03-12 09:46:36 DBG table `main`.`call_center_tmp` dropped
2024-03-12 09:46:36 DBG create table if not exists `main`.`call_center_tmp` (`_sling_row_id` varchar(65533),
`cc_call_center_sk` bigint,
`cc_call_center_id` varchar(65533),
`cc_rec_start_date` datetime,
`cc_rec_end_date` datetime,
`cc_closed_date_sk` varchar(65533),
`cc_open_date_sk` bigint,
`cc_name` varchar(65533),
`cc_class` varchar(65533),
`cc_employees` bigint,
`cc_sq_ft` bigint,
`cc_hours` varchar(65533),
`cc_manager` varchar(65533),
`cc_mkt_id` bigint,
`cc_mkt_class` varchar(65533),
`cc_mkt_desc` varchar(65533),
`cc_market_manager` varchar(65533),
`cc_division` bigint,
`cc_division_name` varchar(65533),
`cc_company` bigint,
`cc_company_name` varchar(65533),
`cc_street_number` bigint,
`cc_street_name` varchar(65533),
`cc_street_type` varchar(65533),
`cc_suite_number` varchar(65533),
`cc_city` varchar(65533),
`cc_county` varchar(65533),
`cc_state` varchar(65533),
`cc_zip` bigint,
`cc_country` varchar(65533),
`cc_gmt_offset` decimal(24,6),
`cc_tax_percentage` decimal(24,6))  distributed by hash(`_sling_row_id`)
2024-03-12 09:46:36 INF streaming data
2024-03-12 09:46:36 INF importing into StarRocks via stream load
2024-03-12 09:46:36 DBG writing to /var/folders/xn/_h8gfpfs3vv_m6gkxm8t0l380000gn/T/starrocks/main/call_center_tmp/2024-03-12T094636.922/part.01 [fileRowLimit=250000 fileBytesLimit=0 compression= concurrency=7 useBufferedStream=false fileFormat=csv]
2024-03-12 09:46:36 DBG Done submitting data. Waiting for load completion.
2024-03-12 09:46:36 DBG loading /var/folders/xn/_h8gfpfs3vv_m6gkxm8t0l380000gn/T/starrocks/main/call_center_tmp/2024-03-12T094636.922/part.01.0001.csv [2.2 kB] ds.1710261996847.W3D-0
2024-03-12 09:46:36 WRN StarRocks redirected the API call to 'http://localhost:8040'. Please use that as your FE url.
2024-03-12 09:46:36 DBG stream-load completed for /var/folders/xn/_h8gfpfs3vv_m6gkxm8t0l380000gn/T/starrocks/main/call_center_tmp/2024-03-12T094636.922/part.01.0001.csv => {
    "TxnId": 38,
    "Label": "b2753334-55a1-40a8-87cf-ff2f0b833f1e",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 6,
    "NumberLoadedRows": 6,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 2201,
    "LoadTimeMs": 25,
    "BeginTxnTimeMs": 0,
    "StreamLoadPlanTimeMs": 1,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 6,
    "CommitAndPublishTimeMs": 17
}
2024-03-12 09:46:36 DBG select count(*) cnt from `main`.`call_center_tmp`
2024-03-12 09:46:36 DBG comparing checksums []string{"_sling_row_id", "cc_call_center_sk", "cc_call_center_id", "cc_rec_start_date", "cc_rec_end_date", "cc_closed_date_sk", "cc_open_date_sk", "cc_name", "cc_class", "cc_employees", "cc_sq_ft", "cc_hours", "cc_manager", "cc_mkt_id", "cc_mkt_class", "cc_mkt_desc", "cc_market_manager", "cc_division", "cc_division_name", "cc_company", "cc_company_name", "cc_street_number", "cc_street_name", "cc_street_type", "cc_suite_number", "cc_city", "cc_county", "cc_state", "cc_zip", "cc_country", "cc_gmt_offset", "cc_tax_percentage"} vs []string{"cc_call_center_sk", "cc_call_center_id", "cc_rec_start_date", "cc_rec_end_date", "cc_closed_date_sk", "cc_open_date_sk", "cc_name", "cc_class", "cc_employees", "cc_sq_ft", "cc_hours", "cc_manager", "cc_mkt_id", "cc_mkt_class", "cc_mkt_desc", "cc_market_manager", "cc_division", "cc_division_name", "cc_company", "cc_company_name", "cc_street_number", "cc_street_name", "cc_street_type", "cc_suite_number", "cc_city", "cc_county", "cc_state", "cc_zip", "cc_country", "cc_gmt_offset", "cc_tax_percentage", "_sling_row_id"}: []string{"cc_call_center_sk", "cc_call_center_id", "cc_rec_start_date", "cc_rec_end_date", "cc_closed_date_sk", "cc_open_date_sk", "cc_name", "cc_class", "cc_employees", "cc_sq_ft", "cc_hours", "cc_manager", "cc_mkt_id", "cc_mkt_class", "cc_mkt_desc", "cc_market_manager", "cc_division", "cc_division_name", "cc_company", "cc_company_name", "cc_street_number", "cc_street_name", "cc_street_type", "cc_suite_number", "cc_city", "cc_county", "cc_state", "cc_zip", "cc_country", "cc_gmt_offset", "cc_tax_percentage", "_sling_row_id"}
2024-03-12 09:46:36 DBG select sum(abs(`cc_call_center_sk`)) as `cc_call_center_sk`, sum(length(`cc_call_center_id`)) as `cc_call_center_id`, sum(cast((UNIX_TIMESTAMP(`cc_rec_start_date`) * 1000000) as UNSIGNED)) as `cc_rec_start_date`, sum(cast((UNIX_TIMESTAMP(`cc_rec_end_date`) * 1000000) as UNSIGNED)) as `cc_rec_end_date`, sum(length(`cc_closed_date_sk`)) as `cc_closed_date_sk`, sum(abs(`cc_open_date_sk`)) as `cc_open_date_sk`, sum(length(`cc_name`)) as `cc_name`, sum(length(`cc_class`)) as `cc_class`, sum(abs(`cc_employees`)) as `cc_employees`, sum(abs(`cc_sq_ft`)) as `cc_sq_ft`, sum(length(`cc_hours`)) as `cc_hours`, sum(length(`cc_manager`)) as `cc_manager`, sum(abs(`cc_mkt_id`)) as `cc_mkt_id`, sum(length(`cc_mkt_class`)) as `cc_mkt_class`, sum(length(`cc_mkt_desc`)) as `cc_mkt_desc`, sum(length(`cc_market_manager`)) as `cc_market_manager`, sum(abs(`cc_division`)) as `cc_division`, sum(length(`cc_division_name`)) as `cc_division_name`, sum(abs(`cc_company`)) as `cc_company`, sum(length(`cc_company_name`)) as `cc_company_name`, sum(abs(`cc_street_number`)) as `cc_street_number`, sum(length(`cc_street_name`)) as `cc_street_name`, sum(length(`cc_street_type`)) as `cc_street_type`, sum(length(`cc_suite_number`)) as `cc_suite_number`, sum(length(`cc_city`)) as `cc_city`, sum(length(`cc_county`)) as `cc_county`, sum(length(`cc_state`)) as `cc_state`, sum(abs(`cc_zip`)) as `cc_zip`, sum(length(`cc_country`)) as `cc_country`, sum(abs(truncate(`cc_gmt_offset`, 0))) as `cc_gmt_offset`, sum(abs(truncate(`cc_tax_percentage`, 0))) as `cc_tax_percentage`, sum(length(`_sling_row_id`)) as `_sling_row_id` from `main`.`call_center_tmp`
2024-03-12 09:46:37 DBG drop table if exists `main`.`call_center`
2024-03-12 09:46:37 DBG table `main`.`call_center` dropped
2024-03-12 09:46:37 DBG create table if not exists `main`.`call_center` (`_sling_row_id` varchar(65533),
`cc_call_center_sk` bigint,
`cc_call_center_id` varchar(65533),
`cc_rec_start_date` datetime,
`cc_rec_end_date` datetime,
`cc_closed_date_sk` varchar(65533),
`cc_open_date_sk` bigint,
`cc_name` varchar(65533),
`cc_class` varchar(65533),
`cc_employees` bigint,
`cc_sq_ft` bigint,
`cc_hours` varchar(65533),
`cc_manager` varchar(65533),
`cc_mkt_id` bigint,
`cc_mkt_class` varchar(65533),
`cc_mkt_desc` varchar(65533),
`cc_market_manager` varchar(65533),
`cc_division` bigint,
`cc_division_name` varchar(65533),
`cc_company` bigint,
`cc_company_name` varchar(65533),
`cc_street_number` bigint,
`cc_street_name` varchar(65533),
`cc_street_type` varchar(65533),
`cc_suite_number` varchar(65533),
`cc_city` varchar(65533),
`cc_county` varchar(65533),
`cc_state` varchar(65533),
`cc_zip` bigint,
`cc_country` varchar(65533),
`cc_gmt_offset` decimal(24,6),
`cc_tax_percentage` decimal(24,6))  distributed by hash(`_sling_row_id`)
2024-03-12 09:46:37 INF created table `main`.`call_center`
2024-03-12 09:46:37 DBG insert into `main`.`call_center` (`_sling_row_id`, `cc_call_center_sk`, `cc_call_center_id`, `cc_rec_start_date`, `cc_rec_end_date`, `cc_closed_date_sk`, `cc_open_date_sk`, `cc_name`, `cc_class`, `cc_employees`, `cc_sq_ft`, `cc_hours`, `cc_manager`, `cc_mkt_id`, `cc_mkt_class`, `cc_mkt_desc`, `cc_market_manager`, `cc_division`, `cc_division_name`, `cc_company`, `cc_company_name`, `cc_street_number`, `cc_street_name`, `cc_street_type`, `cc_suite_number`, `cc_city`, `cc_county`, `cc_state`, `cc_zip`, `cc_country`, `cc_gmt_offset`, `cc_tax_percentage`) select `_sling_row_id`, `cc_call_center_sk`, `cc_call_center_id`, `cc_rec_start_date`, `cc_rec_end_date`, `cc_closed_date_sk`, `cc_open_date_sk`, `cc_name`, `cc_class`, `cc_employees`, `cc_sq_ft`, `cc_hours`, `cc_manager`, `cc_mkt_id`, `cc_mkt_class`, `cc_mkt_desc`, `cc_market_manager`, `cc_division`, `cc_division_name`, `cc_company`, `cc_company_name`, `cc_street_number`, `cc_street_name`, `cc_street_type`, `cc_suite_number`, `cc_city`, `cc_county`, `cc_state`, `cc_zip`, `cc_country`, `cc_gmt_offset`, `cc_tax_percentage` from `main`.`call_center_tmp`
2024-03-12 09:46:37 DBG inserted rows into `main`.`call_center` from temp table `main`.`call_center_tmp`
2024-03-12 09:46:37 INF inserted 6 rows into `main`.`call_center` in 0 secs [8 r/s]
2024-03-12 09:46:37 DBG drop table if exists `main`.`call_center_tmp`
2024-03-12 09:46:37 DBG table `main`.`call_center_tmp` dropped
2024-03-12 09:46:37 INF execution succeeded

2024-03-12 09:46:37 INF [2 / 24] running stream "main"."catalog_page"
2024-03-12 09:46:37 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:46:37 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:46:37 DBG type is db-db
2024-03-12 09:46:37 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:46:37 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:46:37 INF connecting to source database (duckdb)
2024-03-12 09:46:37 INF connecting to target database (starrocks)
2024-03-12 09:46:37 INF reading from source database
2024-03-12 09:46:37 DBG select * from "main"."catalog_page"
flarco commented 6 months ago

Will take a look

flarco commented 6 months ago

Can you try with interactive=true for DUCKDB? https://docs.slingdata.io/connections/database-connections/duckdb

alberttwong commented 6 months ago 
atwong@Albert-CelerData ~ % duckdb duckdb.db
v0.10.0 20b1486d11
Enter ".help" for usage hints.
D show tables;
┌────────────────────────┐
│          name          │
│        varchar         │
├────────────────────────┤
│ call_center            │
│ catalog_page           │
│ catalog_returns        │
│ catalog_sales          │
│ customer               │
│ customer_address       │
│ customer_demographics  │
│ date_dim               │
│ household_demographics │
│ income_band            │
│ inventory              │
│ item                   │
│ promotion              │
│ reason                 │
│ ship_mode              │
│ store                  │
│ store_returns          │
│ store_sales            │
│ time_dim               │
│ warehouse              │
│ web_page               │
│ web_returns            │
│ web_sales              │
│ web_site               │
├────────────────────────┤
│        24 rows         │
└────────────────────────┘
D select count(*) from catalog_page;
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│        11718 │
└──────────────┘
D
alberttwong commented 6 months ago 
atwong@Albert-CelerData ~ % cat ~/.sling/env.yaml
# Environment Credentials for Sling CLI

# See https://docs.slingdata.io/sling-cli/environment

connections:
  DUCKDB:
    type: duckdb
    instance: /Users/atwong/duckdb.db
    interactive: true

  MYSQL:
    type: mysql
    url: mysql://admin:admin@hw2aq4jfy.cloud-app.celerdata.com:9030/albert

  MYSQLLOCAL:
    type: mysql
    url: mysql://root:password@localhost:3306/employees

  POSTGRESLOCAL:
    type: postgres
    url: postgresql://postgres:postgres@localhost:5432/dvdrental?sslmode=disable

  STARROCKSLOCAL:
    type: starrocks
    fe_url: http://localhost:8030
    url: starrocks://root:@localhost:9030/

variables: {}
flarco commented 6 months ago

Weird... will take a look

alberttwong commented 6 months ago 
atwong@Albert-CelerData ~ % sling run -d -r ./replication.yaml
2024-03-12 09:53:39 DBG processing wildcards for DUCKDB
2024-03-12 09:53:39 DBG getting tables for main.*
2024-03-12 09:53:39 DBG csv stream provided is empty

2024-03-12 09:53:39 INF Sling Replication [0 streams] | DUCKDB -> STARROCKSLOCAL
2024-03-12 09:53:39 INF Sling Replication Completed in 0 secs | DUCKDB -> STARROCKSLOCAL | 0 Successes | 0 Failures

this happens when I still am logged in with duckdb.

alberttwong commented 6 months ago 
atwong@Albert-CelerData ~ % cat replication.yaml
source: DUCKDB
target: STARROCKSLOCAL

# default config options which apply to all streams
defaults:
  mode: full-refresh
  object: main.{stream_table}

streams:
  main.*:
atwong@Albert-CelerData ~ % sling run -d -r ./replication.yaml
2024-03-12 09:55:00 DBG processing wildcards for DUCKDB
2024-03-12 09:55:00 DBG getting tables for main.*
2024-03-12 09:55:00 INF Sling Replication [24 streams] | DUCKDB -> STARROCKSLOCAL

2024-03-12 09:55:00 INF [1 / 24] running stream "main"."call_center"
2024-03-12 09:55:00 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:00 DBG type is db-db
2024-03-12 09:55:00 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:00 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:00 INF connecting to source database (duckdb)
2024-03-12 09:55:00 INF connecting to target database (starrocks)
2024-03-12 09:55:00 INF reading from source database
2024-03-12 09:55:00 DBG csv stream provided is empty
2024-03-12 09:55:00 INF execution failed

2024-03-12 09:55:00 INF [2 / 24] running stream "main"."catalog_page"
2024-03-12 09:55:00 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:00 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:00 DBG type is db-db
2024-03-12 09:55:00 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:00 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:00 INF connecting to source database (duckdb)
2024-03-12 09:55:00 INF connecting to target database (starrocks)
2024-03-12 09:55:00 INF reading from source database
2024-03-12 09:55:00 DBG csv stream provided is empty
2024-03-12 09:55:00 INF execution failed

2024-03-12 09:55:01 INF [3 / 24] running stream "main"."catalog_returns"
2024-03-12 09:55:01 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:01 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:01 DBG type is db-db
2024-03-12 09:55:01 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:01 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:01 INF connecting to source database (duckdb)
2024-03-12 09:55:01 INF connecting to target database (starrocks)
2024-03-12 09:55:01 INF reading from source database
2024-03-12 09:55:01 DBG csv stream provided is empty
2024-03-12 09:55:01 INF execution failed

2024-03-12 09:55:01 INF [4 / 24] running stream "main"."catalog_sales"
2024-03-12 09:55:01 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:01 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:01 DBG type is db-db
2024-03-12 09:55:01 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:01 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:01 INF connecting to source database (duckdb)
2024-03-12 09:55:01 INF connecting to target database (starrocks)
2024-03-12 09:55:01 INF reading from source database
2024-03-12 09:55:01 DBG csv stream provided is empty
2024-03-12 09:55:01 INF execution failed

2024-03-12 09:55:01 INF [5 / 24] running stream "main"."customer"
2024-03-12 09:55:01 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:01 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:01 DBG type is db-db
2024-03-12 09:55:01 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:01 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:01 INF connecting to source database (duckdb)
2024-03-12 09:55:01 INF connecting to target database (starrocks)
2024-03-12 09:55:01 INF reading from source database
2024-03-12 09:55:01 DBG csv stream provided is empty
2024-03-12 09:55:01 INF execution failed

2024-03-12 09:55:01 INF [6 / 24] running stream "main"."customer_address"
2024-03-12 09:55:01 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:01 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:01 DBG type is db-db
2024-03-12 09:55:01 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:01 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:01 INF connecting to source database (duckdb)
2024-03-12 09:55:01 INF connecting to target database (starrocks)
2024-03-12 09:55:01 INF reading from source database
2024-03-12 09:55:01 DBG csv stream provided is empty
2024-03-12 09:55:01 INF execution failed

2024-03-12 09:55:01 INF [7 / 24] running stream "main"."customer_demographics"
2024-03-12 09:55:01 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:01 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:01 DBG type is db-db
2024-03-12 09:55:01 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:01 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:01 INF connecting to source database (duckdb)
2024-03-12 09:55:01 INF connecting to target database (starrocks)
2024-03-12 09:55:01 INF reading from source database
2024-03-12 09:55:01 DBG csv stream provided is empty
2024-03-12 09:55:01 INF execution failed

2024-03-12 09:55:01 INF [8 / 24] running stream "main"."date_dim"
2024-03-12 09:55:01 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:01 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:01 DBG type is db-db
2024-03-12 09:55:01 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:01 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:01 INF connecting to source database (duckdb)
2024-03-12 09:55:02 INF connecting to target database (starrocks)
2024-03-12 09:55:02 INF reading from source database
2024-03-12 09:55:02 DBG csv stream provided is empty
2024-03-12 09:55:02 INF execution failed

2024-03-12 09:55:02 INF [9 / 24] running stream "main"."household_demographics"
2024-03-12 09:55:02 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:02 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:02 DBG type is db-db
2024-03-12 09:55:02 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:02 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:02 INF connecting to source database (duckdb)
2024-03-12 09:55:02 INF connecting to target database (starrocks)
2024-03-12 09:55:02 INF reading from source database
2024-03-12 09:55:02 DBG csv stream provided is empty
2024-03-12 09:55:02 INF execution failed

2024-03-12 09:55:02 INF [10 / 24] running stream "main"."income_band"
2024-03-12 09:55:02 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:02 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:02 DBG type is db-db
2024-03-12 09:55:02 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:02 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:02 INF connecting to source database (duckdb)
2024-03-12 09:55:02 INF connecting to target database (starrocks)
2024-03-12 09:55:02 INF reading from source database
2024-03-12 09:55:02 DBG csv stream provided is empty
2024-03-12 09:55:02 INF execution failed

2024-03-12 09:55:02 INF [11 / 24] running stream "main"."inventory"
2024-03-12 09:55:02 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:02 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:02 DBG type is db-db
2024-03-12 09:55:02 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:02 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:02 INF connecting to source database (duckdb)
2024-03-12 09:55:02 INF connecting to target database (starrocks)
2024-03-12 09:55:02 INF reading from source database
2024-03-12 09:55:02 DBG csv stream provided is empty
2024-03-12 09:55:02 INF execution failed

2024-03-12 09:55:02 INF [12 / 24] running stream "main"."item"
2024-03-12 09:55:02 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:02 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:02 DBG type is db-db
2024-03-12 09:55:02 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:02 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:02 INF connecting to source database (duckdb)
2024-03-12 09:55:02 INF connecting to target database (starrocks)
2024-03-12 09:55:02 INF reading from source database
2024-03-12 09:55:02 DBG csv stream provided is empty
2024-03-12 09:55:02 INF execution failed

2024-03-12 09:55:02 INF [13 / 24] running stream "main"."promotion"
2024-03-12 09:55:02 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:02 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:02 DBG type is db-db
2024-03-12 09:55:02 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:02 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:02 INF connecting to source database (duckdb)
2024-03-12 09:55:02 INF connecting to target database (starrocks)
2024-03-12 09:55:02 INF reading from source database
2024-03-12 09:55:03 DBG csv stream provided is empty
2024-03-12 09:55:03 INF execution failed

2024-03-12 09:55:03 INF [14 / 24] running stream "main"."reason"
2024-03-12 09:55:03 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:03 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:03 DBG type is db-db
2024-03-12 09:55:03 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:03 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:03 INF connecting to source database (duckdb)
2024-03-12 09:55:03 INF connecting to target database (starrocks)
2024-03-12 09:55:03 INF reading from source database
2024-03-12 09:55:03 DBG csv stream provided is empty
2024-03-12 09:55:03 INF execution failed

2024-03-12 09:55:03 INF [15 / 24] running stream "main"."ship_mode"
2024-03-12 09:55:03 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:03 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:03 DBG type is db-db
2024-03-12 09:55:03 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:03 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:03 INF connecting to source database (duckdb)
2024-03-12 09:55:03 INF connecting to target database (starrocks)
2024-03-12 09:55:03 INF reading from source database
2024-03-12 09:55:03 DBG csv stream provided is empty
2024-03-12 09:55:03 INF execution failed

2024-03-12 09:55:03 INF [16 / 24] running stream "main"."store"
2024-03-12 09:55:03 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:03 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:03 DBG type is db-db
2024-03-12 09:55:03 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:03 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:03 INF connecting to source database (duckdb)
2024-03-12 09:55:03 INF connecting to target database (starrocks)
2024-03-12 09:55:03 INF reading from source database
2024-03-12 09:55:03 DBG csv stream provided is empty
2024-03-12 09:55:03 INF execution failed

2024-03-12 09:55:03 INF [17 / 24] running stream "main"."store_returns"
2024-03-12 09:55:03 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:03 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:03 DBG type is db-db
2024-03-12 09:55:03 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:03 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:03 INF connecting to source database (duckdb)
2024-03-12 09:55:03 INF connecting to target database (starrocks)
2024-03-12 09:55:03 INF reading from source database
2024-03-12 09:55:03 DBG csv stream provided is empty
2024-03-12 09:55:03 INF execution failed

2024-03-12 09:55:03 INF [18 / 24] running stream "main"."store_sales"
2024-03-12 09:55:03 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:03 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:03 DBG type is db-db
2024-03-12 09:55:03 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:03 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:03 INF connecting to source database (duckdb)
2024-03-12 09:55:03 INF connecting to target database (starrocks)
2024-03-12 09:55:03 INF reading from source database
2024-03-12 09:55:03 DBG csv stream provided is empty
2024-03-12 09:55:03 INF execution failed

2024-03-12 09:55:03 INF [19 / 24] running stream "main"."time_dim"
2024-03-12 09:55:03 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:03 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:03 DBG type is db-db
2024-03-12 09:55:03 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:03 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:03 INF connecting to source database (duckdb)
2024-03-12 09:55:04 INF connecting to target database (starrocks)
2024-03-12 09:55:04 INF reading from source database
2024-03-12 09:55:04 DBG csv stream provided is empty
2024-03-12 09:55:04 INF execution failed

2024-03-12 09:55:04 INF [20 / 24] running stream "main"."warehouse"
2024-03-12 09:55:04 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:04 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:04 DBG type is db-db
2024-03-12 09:55:04 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:04 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:04 INF connecting to source database (duckdb)
2024-03-12 09:55:04 INF connecting to target database (starrocks)
2024-03-12 09:55:04 INF reading from source database
2024-03-12 09:55:04 DBG csv stream provided is empty
2024-03-12 09:55:04 INF execution failed

2024-03-12 09:55:04 INF [21 / 24] running stream "main"."web_page"
2024-03-12 09:55:04 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:04 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:04 DBG type is db-db
2024-03-12 09:55:04 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:04 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:04 INF connecting to source database (duckdb)
2024-03-12 09:55:04 INF connecting to target database (starrocks)
2024-03-12 09:55:04 INF reading from source database
2024-03-12 09:55:04 DBG csv stream provided is empty
2024-03-12 09:55:04 INF execution failed

2024-03-12 09:55:04 INF [22 / 24] running stream "main"."web_returns"
2024-03-12 09:55:04 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:04 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:04 DBG type is db-db
2024-03-12 09:55:04 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:04 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:04 INF connecting to source database (duckdb)
2024-03-12 09:55:04 INF connecting to target database (starrocks)
2024-03-12 09:55:04 INF reading from source database
2024-03-12 09:55:04 DBG csv stream provided is empty
2024-03-12 09:55:04 INF execution failed

2024-03-12 09:55:04 INF [23 / 24] running stream "main"."web_sales"
2024-03-12 09:55:04 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:04 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:04 DBG type is db-db
2024-03-12 09:55:04 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:04 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:04 INF connecting to source database (duckdb)
2024-03-12 09:55:04 INF connecting to target database (starrocks)
2024-03-12 09:55:04 INF reading from source database
2024-03-12 09:55:04 DBG csv stream provided is empty
2024-03-12 09:55:04 INF execution failed

2024-03-12 09:55:04 INF [24 / 24] running stream "main"."web_site"
2024-03-12 09:55:04 DBG could not insert execution into local .sling.db. UNIQUE constraint failed: executions.exec_id
2024-03-12 09:55:04 DBG Sling version: 1.1.13 (darwin arm64)
2024-03-12 09:55:04 DBG type is db-db
2024-03-12 09:55:04 DBG using source options: {"empty_as_null":true,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-03-12 09:55:04 DBG using target options: {"datetime_format":"auto","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-03-12 09:55:04 INF connecting to source database (duckdb)
2024-03-12 09:55:04 INF connecting to target database (starrocks)
2024-03-12 09:55:04 INF reading from source database
2024-03-12 09:55:04 DBG csv stream provided is empty
2024-03-12 09:55:04 INF execution failed

2024-03-12 09:55:05 INF Sling Replication Completed in 4 secs | DUCKDB -> STARROCKSLOCAL | 0 Successes | 24 Failures

fatal:
--- sling_cli.go:384 func1 ---
--- sling_cli.go:430 cliInit ---
--- cli.go:284 CliProcess ---
~ failure running replication (see docs @ https://docs.slingdata.io/sling-cli)
--- sling_logic.go:195 processRun ---

--------------------------- "main"."call_center" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."call_center". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."catalog_page" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."catalog_page". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."catalog_returns" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."catalog_returns". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."catalog_sales" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."catalog_sales". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."customer" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."customer". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."customer_address" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."customer_address". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."customer_demographics" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."customer_demographics". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."date_dim" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."date_dim". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."household_demographics" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."household_demographics". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."income_band" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."income_band". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."inventory" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."inventory". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."item" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."item". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."promotion" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."promotion". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."reason" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."reason". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."ship_mode" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."ship_mode". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."store" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."store". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."store_returns" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."store_returns". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."store_sales" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."store_sales". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."time_dim" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."time_dim". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."warehouse" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."warehouse". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."web_page" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."web_page". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."web_returns" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."web_returns". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."web_sales" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."web_sales". Perhaps it does not exists, or user does not have read permission.

--------------------------- "main"."web_site" ---------------------------
--- task_run.go:98 func1 ---
--- task_run.go:502 runDbToDb ---
--- task_run_read.go:59 ReadFromDB ---
~ execution failed
--- database.go:1490 GetSQLColumns ---
~ Could not ReadFromDB
--- database.go:1624 GetColumns ---
~ Could not get source columns
--- database.go:1555 GetTableColumns ---
did not find any columns for "main"."web_site". Perhaps it does not exists, or user does not have read permission.

I tried with and without FE URL

atwong@Albert-CelerData ~ % cat ~/.sling/env.yaml
# Environment Credentials for Sling CLI

# See https://docs.slingdata.io/sling-cli/environment

connections:
  DUCKDB:
    type: duckdb
    instance: /Users/atwong/duckdb.db
    interactive: true

  MYSQL:
    type: mysql
    url: mysql://admin:admin@hw2aq4jfy.cloud-app.celerdata.com:9030/albert

  MYSQLLOCAL:
    type: mysql
    url: mysql://root:password@localhost:3306/employees

  POSTGRESLOCAL:
    type: postgres
    url: postgresql://postgres:postgres@localhost:5432/dvdrental?sslmode=disable

  STARROCKSLOCAL:
    type: starrocks
#    fe_url: http://localhost:8030
    url: starrocks://root:@localhost:9030/

variables: {}
flarco commented 6 months ago

Fixed. See https://github.com/slingdata-io/sling-cli/issues/216#issuecomment-1994407872 Closing

alberttwong commented 6 months ago

confirmed to work

atwong@Albert-CelerData ~ % cat ~/.sling/env.yaml
# Environment Credentials for Sling CLI

# See https://docs.slingdata.io/sling-cli/environment

connections:
  DUCKDB:
    type: duckdb
    instance: /Users/atwong/duckdb.db
#    interactive: true

  MYSQL:
    type: mysql
    url: mysql://admin:admin@hw2aq4jfy.cloud-app.celerdata.com:9030/albert

  MYSQLLOCAL:
    type: mysql
    url: mysql://root:password@localhost:3306/employees

  POSTGRESLOCAL:
    type: postgres
    url: postgresql://postgres:postgres@localhost:5432/dvdrental?sslmode=disable

  STARROCKSLOCAL:
    type: starrocks
    fe_url: http://localhost:8030
    url: starrocks://root:@localhost:9030/

variables: {}
atwong@Albert-CelerData ~ % cat replication.yaml
source: DUCKDB
target: STARROCKSLOCAL

# default config options which apply to all streams
defaults:
  mode: full-refresh
  object: 'main.{stream_table}'
  source_options:
    limit: 50000

streams:
  main.*: