search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
398 stars 27 forks source link

Schema change detection #257

Closed alberttwong closed 4 months ago

alberttwong commented 5 months ago

Feature Description

Schema change detection. Of course, it'll fail so the question is many continuous testing?

bvallier commented 5 months ago

Is this related to full-refresh mode where an existing schema is present and a new one (e.g. an additional column added) is being materialized? @flarco the default behavior seems to fail in this case, which i didn't expect. I thought the table would be wiped and any new columns would be added to new table, but that didn't seem to be happening. Is this expected?

flarco commented 5 months ago

hey Brice, if it is full refresh, it should recreate the table with the new column. If incremental, it should add the new column (and not fail). Can you paste your log output with debug mode here?

bvallier commented 5 months ago

my apologies - looking closer, this is a dagster issue using SnowflakePandasIOManager...