Open Alp-Edeka opened 1 month ago
Can you try this as a workaround:
servers:
test:
type: dataframe
models:
test_table:
description: Test description.
type: table
fields:
test_field:
required: true
description: Another description.
type: array
title: Test
required: false
example: '[''02'',''03'']'
items:
type: string
description: Last description.
config:
databricksType: array<string>
Thank you for your response @jochenchrist. Unfortunately, I am still getting the same type mismatch. I am assuming that adding something here could fix my issue:
OK, need to dig deeper in here (https://github.com/datacontract/datacontract-cli/blob/7def89252057a6055a6b154fdfd3f14419767a85/datacontract/export/sql_type_converter.py#L114 should respect the config option).
Just to make sure: Are you using the latest version of the CLI tool?
@jochenchrist I am using version 0.10.7, not the latest one.
"More simple" data types seem to also have the same issue. For example, the data contract
servers:
test:
type: dataframe
models:
test_model:
description: Test description 1.
type: table
fields:
test_field:
required: true
description: Test description 2.
type: timestamp_tz
example: "2024-06-01T12:00:00.000Z"
config:
databricksType: timestamp
throws the output:
Column,Event,Details
test_field,:icon-fail: Type Mismatch, Expected Type: timestamp_tz; Actual Type: timestamp
Just to get sure, could you try testing with latest version v0.10.9
?
(you might need to install with extras pip install datacontract-cli[all] --upgrade
)
I now tried
%pip install datacontract-cli[all] --upgrade
dbutils.library.restartPython()
inside a notebook and ran the test again with the same outcome.
I am using Databricks and trying to test fields in my table that include arrays. My contract is as follows:
My table is defined as:
Now using datacontract-cli inside Databricks throws the following output and my contract fails:
How can I actually check those type of fields inside Databricks?