search

Amsterdam / dq-suite-amsterdam

The DQ suite should ease the use of Great Expectations in the context of the Gemeente Amsterdam
1 stars 2 forks source link

Enable writing new delta tables #54

Open SSchotten opened 2 months ago

SSchotten commented 2 months ago

This PR should address the current error: 

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dq_suite/common.py:198, in merge_df_with_unity_table(df, catalog_name, table_name, table_merge_id, df_merge_id, merge_dict, spark_session)
    194 full_table_name = get_full_table_name(
    195     catalog_name=catalog_name, table_name=table_name
    196 )
    197 df_alias = f'{table_name}_df'
--> 198 regelTabel = DeltaTable.forName(spark_session, full_table_name)

....

AnalysisException: [DELTA_MISSING_DELTA_TABLE] `data_quality`.`brondataset` is not a Delta table.