on_schema_change not working in iceberg incremental models

[e-quili]

Describe the bug

The parameter on_schema_change, which configures behavior when the schema changes in an incremental model, seems to be ignored with iceberg tables.

Current behavior

When we add the column "new_col" to an existing incremental model configured as below, the new column is not added to the materialized table.

{{ 
    config(
        materialized='incremental',
        incremental_strategy='merge',
        unique_key='table_id',
        on_schema_change='append_new_columns',
        file_format= 'iceberg'
    )
}} 

SELECT  
        CAST(table_id AS string) AS table_id 
       ,CAST(value AS float) AS value
       ,CAST(1 AS int) AS new_col   -- new column added afterwards
FROM 
       glue_catalog.{{ ref('test_table') }} 

Expected behavior

With the setting on_schema_change='append_new_columns', "new_col" should be added to the materialized table.

Screenshots and log output

• incremental_strategy='merge' : we don't get any error message
• incremental_strategy='append' : we get the error message "too many data columns"

System information

The output of dbt --version:

Core:
  - installed: 1.8.6
  - latest:    1.8.6 - Up to date!

Plugins:
  - glue:   1.8.1 - Up to date!
  - spark:  1.8.0 - Up to date!

The operating system you're using: Windows

The output of python --version: Python 3.12.6