airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.76k stars 3.79k forks source link

Low code connector YAML is missing a proper JSONSchema in documentation #32860

Open tonyxiao opened 7 months ago

tonyxiao commented 7 months ago

Page URL

https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview

Description

Custom config files implemented in JSON would benefit a lot from having proper JSONSchema. This makes auto-complete and "type checking" work automatically in things like VSCode. Projects such as turbo & openAPI all have them. But Airbyte's low code spec seems to be missing

Examples of other schemas https://turbo.build/schema.json https://json-schema.org/draft/2020-12/schema https://spec.openapis.org/oas/3.1/dialect/base

How they can be used turbo.json

{
  "$schema": "https://turbo.build/schema.json",
    "clean": {},
    "build": {
      "dependsOn": ["^build", "clean"]
    }
  }
}
tonyxiao commented 7 months ago

Apparently it is located here https://docs.airbyte.com./assets/files/declarative_component_schema-f15b8aaab5a96fa0ceef85db5cab8b74.yaml

But we should add it to the docs and make it WAY more accessible. The default YAML export should contain a $schema: https://docs.airbyte.com./assets/files/declarative_component_schema-f15b8aaab5a96fa0ceef85db5cab8b74.yaml as well.

flash1293 commented 7 months ago

This is a link to the most recent version (note that new features might be added over time): https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml

Still agree with what has been said - we should add the reference to the generated YAML file and also link it in the documentation.

Pinging @airbytehq/connector-extensibility

lmossman commented 7 months ago

We already reference the schema and link it in the docs here: https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference

@tonyxiao is this what you are looking for?

tonyxiao commented 7 months ago

Ah I see. I missed it. Maybe it should be more visible / available in the overview section?

girarda commented 7 months ago

I'm on board with both suggestions. As a quick fix, we can add a link from the overview to the schema

tonyxiao commented 7 months ago

fyi for yaml i think this is the syntax that works best for vscode.

# yaml-language-server: $schema=https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.json # i think needs to be json as vscode doesn't seem to work with this as `yaml` somehow. 
... rest of the yaml