Open yu-iskw opened 1 year ago
Agreed! I don't think that JSON Schema itself is extensible in that way, but if there is any way to augment a schema with components from another schema then that would be awesome.
@joellabes I got an idea to support other tools which define their custom meta by utilizing $ref
to include external files and anyOf
to unify them. We can independently define custom meta for each tool and include them in dbt_yml_files-latest.json
.
To enhance interoperability between dbt and tools like Lightdash, I propose extending the JSON schemas used in dbt to support external schema references. This approach will allow seamless integration and customization of metadata at various levels, facilitating better data management and analysis.
Use the $ref feature to include external schemas, enabling separate management of schemas while maintaining a unified structure. Merging Definitions with anyOf:
Support for Various Resource Types:
Resource types to be supported:
Examples of External Schemas for Lightdash:
{
"models": {
"type": "array",
"items": {
"type": "object",
"required": ["name"],
"properties": {
"name": { "type": "string" },
"description": { "type": "string" },
"access": {
"type": "string",
"enum": ["private", "protected", "public"]
},
"columns": {
"type": "array",
"items": {
"$ref": "#/$defs/column_properties"
}
},
"config": { "$ref": "#/$defs/model_configs" },
"constraints": { "$ref": "#/$defs/constraints" },
"data_tests": {
"type": "array",
"items": { "$ref": "#/$defs/data_tests" }
},
"deprecation_date": { "type": "string" },
"docs": { "$ref": "#/$defs/docs_config" },
"group": { "$ref": "#/$defs/group" },
"latest_version": { "type": "number" },
"meta": {
"anyOf": [
{ "type": "object" },
{ "$ref": "./meta/lightdash/model_table_meta.json" }
]
},
"tests": {
"type": "array",
"items": { "$ref": "#/$defs/data_tests" }
},
"versions": {
"type": "array",
"items": {
"type": "object",
"required": ["v"],
"properties": {
"columns": {
"type": "array",
"items": {
"anyOf": [
{ "$ref": "#/$defs/include_exclude" },
{ "$ref": "#/$defs/column_properties" },
{ "$ref": "./meta/lightdash/model_column_meta.json" }
]
}
},
"config": { "$ref": "#/$defs/model_configs" },
"v": { "type": "number" }
}
}
}
},
"additionalProperties": false
}
}
}
Motivation
Some tools which integrate with dbt defines their own custom schema under the
meta
property at table-level and column-level. For instance, lightdash enables us to declare metrics like below.https://docs.lightdash.com/guides/how-to-create-metrics
We define the
meta
property just as object. I don't have any good ideas to support such extensibility in JSON schema. But, if we make the JSON schema opened to other tools, that would be awesome.https://github.com/dbt-labs/dbt-jsonschema/blob/main/schemas/dbt_yml_files.json#L680-L682