slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
299 stars 16 forks source link

Defaults Object not being applied correctly when Wildcard is used #308

Closed flarco closed 3 weeks ago

flarco commented 1 month ago

Config:

source: S3
target: SNOWFLAKE_DB

defaults:
  mode: full-refresh
  object: '{target_schema}.{stream_file_name}'

  source_options:
    format: csv
    empty_as_null: true

  target_options:
    column_casing: snake

streams:
  "s3://bucket_name/data/PARSED_FILES/FILE_LOCATION1/STARTS_WITH_*.csv":
    object: 'raw.file_location1' # overwrite default object

  "s3://bucket_name/data/PARSED_FILES/FILE_LOCATION2/STARTS_WITH_*.csv":
    object: 'raw.file_location2' # overwrite default object

  "s3://bucket_name/data/PARSED_FILES/FILE_LOCATION3/STARTS_WITH_*.csv":
    object: 'raw.file_location3' # overwrite default object

env:
  SLING_LOADED_AT_COLUMN: true
flarco commented 3 weeks ago

Fixed in #303