search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
398 stars 27 forks source link

allow errors when loading csv files into StarRocks #256

Closed rishabhkaushal07 closed 5 months ago

rishabhkaushal07 commented 5 months ago

Feature Description

Hi, StarRocks has a feature where you can set error tolerance when loading a csv file into a starrocks db.

sling does not have this feature and will error out if one of the rows of a CSV file has errors or can't be loaded. It would be good to have this feature as it would still ingestion of the rows without out right failing.

flarco commented 5 months ago

Hi, done: https://github.com/slingdata-io/sling-cli/pull/253/commits/6a7495ae3578d32845348188199edb3ede4794b5

Will just have to set as a connection property:

export STARROCKS='{ type: starrocks,
  url: "starrocks://myuser:mypass@host.ip:9030/mydatabase",
  fe_url: "http://host.ip:8030",  max_filter_ratio: "0.3", strict_mode: "true" }'