dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
401 stars 228 forks source link

[Bug] dbt docs generate does not include Data Type (data stored in iceberg format) #988

Open troylepnc opened 8 months ago

troylepnc commented 8 months ago

Is this a new bug in dbt-spark?

Current Behavior

"dbt docs generate" does not pick up Data Type; column Type is empty

image

Expected Behavior

Column Type shows data type information as specified in dbt model properties (yaml)

Steps To Reproduce

Dbt command: dbt docs generate

models:

  • name: my_test_model description: 'my test model' meta: owner: acme columns:
  • name: id data_type: bigint description: 'Internally managed primary key' constraints:
  • type: not_null
  • name: loadcommandid data_type: bigint description: 'Parent.' constraints:
  • type: not_null
  • name: parametername data_type: string description: 'Parameter name.' constraints:
  • type: not_null
  • name: parametervalue data_type: string description: 'Parameter value.'
  • name: publicid data_type: string description: 'ID or primary key of the row' constraints:
  • type: not_null

Relevant log output

No response

Environment

- OS: Windows 11
- Python: 3.12.1
- Spark: spark-3.5.0-bin-hadoop3-scala2.13
- dbt-core: 1.7.7
- dbt-spark: 1.7.1
- apache iceberg: 1.4.3

Additional Context

"dbt run" and "dbt docs generate" complete successfully with no errors, but data type (column Type) is showing blank in docs output

nyc-de commented 4 months ago

I ran into this as well. I randomly tried replacing the data_type key with type and it worked.

In the screenshot you can see the docs generate is picking up the column type where the key is replaced.

Screenshot 2024-06-20 at 2 18 11 PM Screenshot 2024-06-20 at 2 18 25 PM
mabubakr007 commented 3 months ago

I ran into this as well. I randomly tried replacing the data_type key with type and it worked.

In the screenshot you can see the docs generate is picking up the column type where the key is replaced.

Screenshot 2024-06-20 at 2 18 11 PM Screenshot 2024-06-20 at 2 18 25 PM

Thanks for this. It worked for us