slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
301 stars 16 forks source link

If you define database but don't define the table, Sling doesn't target replicate #218

Closed alberttwong closed 3 months ago

alberttwong commented 3 months ago
atwong@Albert-CelerData ~ % sling run --src-conn DUCKDB  --src-stream 'duckdb.main.call_center' --tgt-conn STARROCKSLOCAL --tgt-object 'testing' --mode full-refresh
9:21AM WRN Could not successfully get format values. Blank values for: object_schema
9:21AM INF connecting to source database (duckdb)
9:21AM INF connecting to target database (starrocks)
9:21AM INF reading from source database
9:21AM WRN Could not successfully get format values. Blank values for: object_schema
9:21AM INF writing to target database [mode: full-refresh]
^C
interrupting...
^C^C

^C^C
^C^C^C9:25AM INF execution failed
atwong@Albert-CelerData ~ % sling run --src-conn DUCKDB  --src-stream 'duckdb.main.call_center' --tgt-conn STARROCKSLOCAL --tgt-object 'testing.call_center' --mode full-refresh
9:25AM INF connecting to source database (duckdb)
9:25AM INF connecting to target database (starrocks)
9:25AM INF reading from source database
9:25AM INF writing to target database [mode: full-refresh]
9:25AM INF streaming data
9:25AM INF importing into StarRocks via stream load
9:25AM WRN StarRocks redirected the API call to 'http://localhost:8040'. Please use that as your FE url.
9:25AM INF created table `testing`.`call_center`
9:25AM INF inserted 6 rows into `testing`.`call_center` in 1 secs [5 r/s]
9:25AM INF execution succeeded

I had to use target to a defined database.table inorder for it to work.

flarco commented 3 months ago

in your STARROCKSLOCAL connection, do you have a default schema defined (schema=my_schema)? Will take a look, it should use the default schema if defined.

alberttwong commented 3 months ago

yes... i think it will work if I defined the schema. I was just trying to see if source=schema.tablename would target=schema (assume that the table name stays the same).

flarco commented 3 months ago

Fixed by https://github.com/slingdata-io/sling-cli/commit/09abd44a907e929138a439d0b7a136ca8a5daf51 Closing.

flarco commented 3 months ago

It will error asking for a schema in the object. Duckdb was the culprit.