Closed raghuj5222 closed 9 months ago
Hi ! I don't remember dbt-dremio providing option for tmp table creation. Couldn't this be related to :
https://docs.dremio.com/current/reference/sql/commands/apache-iceberg-tables/apache-iceberg-create/
?
Could you please enable the option and retry ?
Best regards from French west coast. Fabrice
Is there an existing issue for this?
Current Behavior
When a dbt model is incremental - dbt creates a _tmp table during processing of the model.
It is not possible to pass the file_format as "iceberg" or set tblproperties -> table='iceberg'. This causes the table to be created as a parquet table and causes an error running the model
dbt tries to read the information_Schema and doesnt find a table in Dremio
step 1. create table "ECS S3"."asset360"."IcebergDesPoc"."staging_pb_volumes_v4__dbt_tmp" as ( select col from tableA)
-- the above creates a parquet rather than iceberg. can we pass config to define it as a iceberg table
step 2. select column_name as column_name ,lower(data_type) as data_type ,character_maximum_length ,numeric_precision ,numeric_scale from information_schema.columns where ilike(table_schema, 'ECS S3.asset360.IcebergDesPoc') and ilike(table_name, 'staging_pb_volumes_v4__dbt_tmp') order by ordinal_position
It expects a table - doesnt find it causing an error
Error:
ERROR: Failure parsing the query. [0m22:44:38.527134 [debug] [Thread-1 ]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '79961fbc-7f5a-4af4-b8c1-8c9834a81f79', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000002C5EE9DB040>]} [0m22:44:38.529159 [error] [Thread-1 ]: 1 of 1 ERROR creating sql incremental model ECS S3.asset360.IcebergDesPoc.staging_pb_volumes_v4 [[31mERROR[0m in 486.59s] [0m22:44:38.530126 [debug] [Thread-1 ]: Finished running node model.naturalgasassets.staging_pb_volumes_v4
Expected Behavior
No response
Steps To Reproduce
{{ config( materialized='incremental', tblproperties= { 'table_type' :'ICEBERG' } ) }}
tblproperties doesnt work
Environment
Relevant log output