This PR addresses the issue related to unique_tmp_tablesuffix when working with Spark. While the proposed changes in issue #668 worked for part of the scenarios, they did not account for Spark’s restrictions on table and view names. Specifically:
• Spark does not allow hyphens (-) in table/view names, causing failures when creating temporary tables.
• This PR modifies the logic to replace hyphens in the generated suffix with underscores (), ensuring Spark compatibility.
Additionally:
• Ensures unique_tmp_table_suffix is respected and functional when set to True for all table types and strategies.
• Supports concurrent runs of the same model by avoiding naming conflicts.
Models used to test - Optional
• incremental materialization tests with Spark, Hive, and Iceberg configurations.
• Test cases included concurrent runs of the same model.
Description
This PR addresses the issue related to unique_tmp_tablesuffix when working with Spark. While the proposed changes in issue #668 worked for part of the scenarios, they did not account for Spark’s restrictions on table and view names. Specifically: • Spark does not allow hyphens (-) in table/view names, causing failures when creating temporary tables. • This PR modifies the logic to replace hyphens in the generated suffix with underscores (), ensuring Spark compatibility.
Additionally: • Ensures unique_tmp_table_suffix is respected and functional when set to True for all table types and strategies. • Supports concurrent runs of the same model by avoiding naming conflicts.
Models used to test - Optional
Checklist