search

MeltanoLabs / Singer-Working-Group

Working group for ongoing development and iteration of the Singer Spec, the de-facto protocol for open source data connectors. Please use "Issues" to create discussion items - or use "Discussions" for general questions.
Apache License 2.0
13 stars 4 forks source link

JSON Schema Version #17

Open visch opened 2 years ago

visch commented 2 years ago

As a new tap/target developer you have to figure this out.

Multiple steps here, one is just to document in the spec which json version is currently supported. Maybe that's the end of it, maybe we go farther and work towards adding a new json version compatibility

visch commented 2 years ago

Not a huge deal for me but it seems like a pretty simple first one to just add that currently the json schema version that is supported is JSON Draft-04

aaronsteers commented 2 years ago

@dmosorast - Do you know if the Singer docs ever declared a JSON Schema Draft as canonical? As @visch points out, I think most people rely on Draft 4 and I think it would be great to document this somewhere in the spec. If it would be desirable to support higher versions of JSON Schema, those capabilities might be exposed in the proposals: #8 and/or #5.

aaronsteers commented 2 years ago

I started researching to better understand backwards compatibility between JSON Schema draft versions:

  1. There are a few listed non-backwards compatible changes between Draft 04 and Draft 06:
  2. Draft 07 is described as follows in the docs: "For the Core and Validation specifications, draft-07 is a relatively minor update. In terms of validation keywords and outcomes, it is fully backwards-compatible with draft-06."

While it seems likely that taps running Draft 04 will emit JSON Schema parseable by targets running other newer versions, I'm not positive that the reverse is true. Does anyone else know?

dmosorast commented 2 years ago

No I don't believe a version of JSON schema has been officially declared anywhere, though we've been going by Draft 4 because it's been working out fine heh. I do agree that stating Draft 4 somewhere would probably be a good idea.

For upgrading the spec, I'm interested in it, but I don't know if there are many use cases that apply to Singer taps. In general the conversation on upgrading hasn't been in terms of a feature of a more recent draft that someone would like to use, but rather to upgrade just because it's newer. As a general best practice, I like to stay away from being too clever in schemas (e.g., extra keywords like maximum/minimum on integers, etc.) because it usually just puts a point of failure in the "EL" process if the source widens their backend fields, for example. Anyway, that's my first morning thoughts here :)

aaronsteers commented 2 years ago

I found this on the Schema Store contribution guidelines and it seems wise/apt for our cases as well:

From the Schema Store best practices:

Use the lowest possible schema draft needed, preferably Draft v4, to ensure interoperability with as many supported editors, IDEs and parsers as possible.

karenetheridge commented 2 years ago

That doesn't seem quite right because a lot of implementations don't support the oldest drafts at all.

If you're sure your schema is compatible across all versions, it would be better to leave out the $schema keyword rather than pin it to an old version.