Data cleaning and/or data quality as part of the sling process

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

https://docs.slingdata.io

GNU General Public License v3.0

301 stars 16 forks source link

Data cleaning and/or data quality as part of the sling process #251

Open alberttwong opened 2 months ago

alberttwong commented 2 months ago

It's kinda the same thing with quality as the check and cleaning is the remediation.

alberttwong commented 2 months ago

Hello, is it possible to LOAD from CSV with variable number of columns, lets say there are files with 10 and 11 columns in them, is it possible to default missing column to NULL, or something and not get an error Value count does not match column count ? Or would I have to do multiple loads with strict=false property?