Open matt-winkler opened 1 year ago
Have you tried with Delta (just to try to scope the bug)? I'm unable to get the error you're seeing, but then again, I'm trying to recreate in the dbt test harness. I get a different error instead, related to missing column, and I get it regardless of file format.
hey @benc-db , We don't see the error on Delta sources. And it's not a column modification issue. Table metadata stays the same. Here is the whole story.
Steps To Reproduce
source
on a parquet table in cloud storage / the lake ( ParquetTable1 )dbt snapshot
(just for initially creating a snapshot table)dbt snapshot
a. dbt-snapshot materialization creates view ParquetTable1__dbt_tmp
b. dbt-snapshot materialization runs a MERGE
statement.. target: ParquetTable1_snapshot
source: ParquetTable__dbt_tmp
(view)between 3.a.
and 3.b.
another process (not in Databricks) re-populates the table ParquetTable1
(with SparkSQL, insert-override). The underlying s3 parquet files are changing as expected. It will be a bit hard to reproduce the issue because we need to modify/remove the s3 file exactly between 3.a.
and 3.b.
and there are a few seconds only :)
more context;
ParquetTable1
is not a partitioned table, refreshed hourly with a non-databricks spark Job.refresh table ParquetTable1
as a pre-hook command to the snapshot, but it didn't help as it ran before 3.a.cc/ @matt-winkler
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.
Describe the bug
When running a dbt
snapshot
on top of an underlying parquet data source, there is a challenge introduced in particular when columns are added / removed at the source. Note that this environment is NOT running with Unity Catalog yet. I'm not sure if that has an impact, but feels relevant to mention.Steps To Reproduce
source
on a parquet table in cloud storage / the lakedbt snapshot
dbt snapshot
again and observe the error message belowNOTE: If
dbt snapshot
is run a THIRD time, it works. This makes it a bit hard to understand the error message's reference to "restarting the cluster", because that doesn't seem to be strictly necessary.Expected behavior
Snapshot works on the second run.
Screenshots and log output
System information
The output of
dbt --version
:The operating system you're using: dbt Cloud