JSON schema

[dontcallmedom]

See http://lists.w3.org/Archives/Public/site-comments/2018Jan/0005.html and https://datatracker.ietf.org/doc/draft-handrews-json-schema/

[tidoust]

A few notes following discussions with Ben Hutton:

• All efforts are being tracked on the JSON Schema Web site
• Upcoming v07 draft of JSON Schema should be ready in a few months from now.
• Many tools still follow draft v04. Goal for standardization is not to rubber-stamp v07, but to find the right stable set of features for JSON Schema.
• JSON Schema's main strength and weakness is that it is used by many many companies and developers around the world, often for "small" schemas. So it's both pervasive and hard to single out a core set of companies that could drive the standardization effort.
• As such, if JSON Schema is to move to standardization, next step would be to assemble a standardization community around it. Idea could be to look at products that embed JSON Schema one way or the other, following the chain of dependencies on npm and others (e.g. the list of projects that depend on the json-schema library or the list of projects that depend on ajv)

[tidoust]

Another point to keep in mind for later, again if JSON Schema is to move to standardization: we would need to sort out copyrights and license for the document. The initial versions of JSON Schema were written by people who are no longer around. Also, I don't see any reference to licensing terms on json-schema.org.

[awwright]

@tidoust As far as licensing goes, as an IETF submission, all parts of the drafts are covered by RFC 5378 (as you're probably familiar, I'm not sure if that's sufficient though). Additionally, all of my work (which is around half of each of the documents) is public domain.

[handrews]

@tidoust I should double-check with my employer, but when I started working on the spec they had no concerns over it, and were familiar with IETF IP rules (other folks in the company have contributed to RFCs). Anything that I would personally have rights to I'm placing in the public domain. I'm pretty sure that 90% of the current draft has been reworked to some degree or another by either @awwright or myself, if that makes any difference. Of the past authors, only Gary Court has avoided replying to any effort to reach him. We've interacted with Krys and Geraint within the last few months, and I think Francis is active in other projects on GitHub.

[Relequestual]

@tidoust Apologies for still having not replied to your follow up email. Joining the W3C from a company perspective for me is not going to happen. I think we're focusing on getting draft-8 done now, but we'll set aside some time to discuss this soon!

[epicfaace]

@Relequestual Has the discussion on whether to move JSON Schema to W3C happened yet? I'm curious to know what the conclusion was.

[iherman]

The JSON Schema homepage has the following remark:

The JSON Schema project intends to shepherd all four draft series to RFC status. Currently, we are continuing to improve our self-published Internet-Drafts. The next step will be to get the drafts adopted by an IETF Working Group. We are actively investigating how to accomplish this.

In other words, it looks like the community has decided to go with the IETF and not with W3C. If this is confirmed, we should probably close this issue.

@tidoust ?

[handrews]

@iherman this has not been resolved one way or the other.

That statement is on the JSON Schema web site because unless/until a concrete new plan is present, there is no reason to take down the prior plan. We're still publishing IETF drafts (hopefully again by the end of this month, in fact).

However, the same issues that brought up the question of IETF vs W3C vs ??? still exist.

[iherman]

Thanks @handrews, good to know!

[epicfaace]

What are the general benefits/drawbacks/considerations for deciding to host a spec with IETF vs W3C?

[akuckartz]

What is the relation/compatibility with JSON-LD ?

[handrews]

@epicfaace it's discussed in detail at https://github.com/json-schema-org/json-schema-spec/issues/526

In particular, as you can see at https://mailarchive.ietf.org/arch/msg/json/w--4QtcF9remEI3FfGM6ghBFkEQ that the folks on the IETF JSON mailing list were very uninterested in the project. There are a couple of competing projects, none of which are widely used or implemented, which the IETF seems to prefer. The fact that many popular projects and products use JSON Schema, and have for many years, and in some cases are actively collaborating with us on the direction of the spec, seems to be completely irrelevant to the IETF folks.

For us at the JSON Schema project, it is important to continue to support our large existing user community. Since the IETF is only interested in starting over, at least for a standards-track RFC, it seems unlikely that we'll go that route. We could publish an informational RFC, though, which might be the fastest track to a "standard" (informational RFCs are, of course, not actually official standards, but they are at least published which is sometimes mostly what people want).

Speaking only for myself, I would expect having JSON Schema adopted by a standards group would result in further changes and refinements- that does not bother me. But I don't see the point in us throwing out what we have entirely and starting over, but still calling it JSON Schema.

[handrews]

@akuckartz I see the technologies as complementary.

JSON Schema essentially lets you do two things: assert conditions about a JSON instance document, and annotate that document with further information. Assertions and annotations can be combined such that annotations are applied conditionally depending on which assertions are found to be valid vs invalid.

You can also use those assertions and annotations without an instance to generate things related to the set of possible instances, such as a web form that can be used to construct an instance, or an OO class that can work with an instance in a particular programming language.

It does not specifically work with semantics (although the format keyword sort-of does, but format is a bit of a mess- it gets slightly better in the next draft, but fundamentally if you want to work with semantics you should go with JSON-LD).

I believe that JSON-LD could be mixed with JSON Schema using JSON Schema's annotation functions. Essentially, JSON Schema's format keyword attempts to annotate fields/objects/arrays in a JSON document with their semantic meaning, but is a very limited way of doing that.

It's been ages since I even looked at JSON-LD so I can't give a good example off the top of my head.

[BigBlueHat]

It's been ages since I even looked at JSON-LD so I can't give a good example off the top of my head.

I'd be happy to help explore making JSON Schema and JSON-LD work (even) better together. The Web of Things WG's Thing Description spec builds upon a sub-set of the JSON Schema vocabulary and integrates JSON-LD alongside, so folks in that community might also be interested in helping there.

[BigBlueHat]

Speaking only for myself, I would expect having JSON Schema adopted by a standards group would result in further changes and refinements- that does not bother me. But I don't see the point in us throwing out what we have entirely and starting over, but still calling it JSON Schema.

JSON Schema's wide adoption (and adoption/use within various W3C specifications) makes it a great candidate for a home at the W3C. The place (afaik) to start would be via a Community Group.

I'd be happy to work with y'all to make that happen--and I've reached out to @Relequestual recently to discuss that (and then was pointed here by @iherman 😄).

I'm subscribed to the https://github.com/json-schema-org/json-schema-spec/issues/526 issue and would be happy to help assist in getting a Community Group started--which should be the right place to sort out IP concerns, setup calls/processes, and get things moving forward under the W3C banner.

There's a lot more tofu making beyond that, but the CG model should be the right place to take the first steps! Or so I hope. 😃

[Relequestual]

My feeling is that we put this discussion on hold till at least half way through 2020. We have our work cut out getting the test suite done for draft 2019-09.

[iherman]

@Relequestual understood.

[BigBlueHat]

@Relequestual do reach out along the way, as I'd be happy to help you all get integrated here!

Also "done" specs don't ever stay "done" when they go through any standards process...so starting earlier than later has massive benefits.

I'll be around in both communities, so please ping me as you need/want help in this area.

❤️, 🎩

[handrews]

@BigBlueHat Part of the reason for waiting is that I've been the one driving the major features like vocabulary support, and those are hard enough within our own little group to get to a point where we can publish and seek feedback.

Given the choice between delaying to involve a formal group vs reaching feature-complete as quickly as possible, when we have major users waiting on the changes who have their own timelines that are more industry-driven than standards-body-driven, my strong preference is to produce a more or less functional, feature-complete spec.

I do not personally have the patience to deal with a formal standards body, while @Relequestual is both willing and better suited to it 😄 I'll probably be actively around for one more significant draft incorporating feedback on the slightly unfinished features, after which I'll step back and the formal standardization process can do whatever they do. I'll be available for consultation as needed, but I will not be looking for any control over where it ends up after that. For me, "done" is "it meets the needs of the existing community who are already trying to use it."

I'll be enthusiastically rooting for the standards process, regardless of whether the end result looks like what goes in or not. But I want to resolve my own work and do a clean hand-off rather than a tug-of-war between visions that will take a near-complete system and re-introduce uncertainty. Which seems inevitable with more stakeholders. I recognize that this is a bit of a selfish position, but I've put a lot into the last several drafts and this is the only way I could finish that without total burnout. I want to hand off something that makes sense as a system, because that seems like the best starting point for the next phase.

And I feel justified in having held off by the imminent inclusion of the latest draft into OpenAPI 3.1, which will resolve an extremely problematic years-long schism (in which there are no villains- everyone has made the best available choices). I'm more concerned with that practical problem than any formal stamp (others definitely have other priorities!).

Anyway, if you're concerned over any delay in engaging the formal standards process, my point here is that I'm the roadblock more than @Relequestual or anyone else is. But I am working to get out of the way. Feel free to reach out to me on the json schema slack if you want to discuss my personal concerns in more detail, otherwise @Relequestual is the person you want to talk with 😁

[pchampin]

I had a conversation today with @egekorkan from Siemens and from the WoT WG (NB: the WoT WG works quite a lot with JSON-Schema, see for example that blog post). He mentioned that JSON-Schema's governance has quite changed since we had the conversation above, and that it might be worth reactivating this debate again.

Note that currently, citing JSON-Schema normatively is challenging, because despite being largely used an implemented, JSON-Schema is not specified by a recognized entity. The WoT WG has encountered this problem, and was led to duplicate part of the JSON-Schema spec in its own REC for that reason -- which is a source of confusion for readers...

[iherman]

FYI, the VC WG has the same problems concerning the normative reference to JSON-Schema; we would also like to use it in our spec.

[lu-zero]

The current specifications are according to here

• draft-bhutton-json-schema-01 (core)
• draft-bhutton-json-schema-validation-01
• draft-bhutton-relative-json-pointer-00

And since its relationship with the WoT DataSchema is not as clear as it should we'll have to clarify and figure out better how to express which is the supported subset. An implementor as today would have to support the full json-schema (current) and still hope for the best to be interoperable.

[iherman]

See also https://github.com/orgs/json-schema-org/discussions/3#discussioncomment-6290893

[lu-zero]

If they want to stay non-standard and evolving, it would be quite problematic to use it in our specifications.

[handrews]

• draft-bhutton-relative-json-pointer-00

Note that Relative JSON Pointer is on a separate standardization path, and I just updated it this month: draft-hha-relative-json-pointer-00. I am talking with some IETF folks about the right path to RFC publication.

For some reason, the HTMLized rendering does not show the link between it and the bhutton-00 draft. I have notified IETF tech support, but you can see the proper relationship in the datatracker.

[mnot]

@handrew what are your next steps for that spec? IETF is in a few weeks and DISPATCH would be the most obvious place to talk about it...

[handrew]

@handrew what are your next steps for that spec? IETF is in a few weeks and DISPATCH would be the most obvious place to talk about it...

@mnot I think you got the wrong guy, brother. Tagging @handrews

[mnot]

Sorry @handrew!

[handrews]

@mnot honestly I'm not sure and am open to suggestion. I had talked a bit with @darrelmiller about it given that Relative JSON Pointer came up in the context of HTTP API-related things. But as he pointed out, it's really not specific to that.

At the moment I'm not planning to actually attend IETF 117, but I live in SF so I'll be around if anyone wants to meet up on the sidelines. If there's a suitably compelling reason I suppose I could spring for a 1-day pass.

From my point of view, the only thing missing is clarification on how to resolve a Relative JSON Pointer against a JSON Pointer to produce another JSON Pointer, without having a document available. There are some cases that are not quite straightforward (such as when using - as an array index, or if the RJP does index manipulation given that without a document it's can't be determined whether that location is an array or not) that need to either be clarified or excluded from such a process.

But I have no idea what else might come up from others, or whether there would be resistance to it that might make it a better candidate for an informational RFC.

Incidentally, I'm also using [Relative] JSON Pointer Templates in a project, which I made up because JSON Path and JMESPath were both frustratingly more complex than I needed and/or not quite what I needed. Not sure if it's worth trying to standardize those or not - they're perhaps more an expression of personal frustration than a necessary addition to the ecosystem. Of course, they would probably better be done separately if at all, and I just mention them for completeness.

I would definitely like to see RJP as an RFC of some sort. I find myself using them very frequently.

[OR13]

Sorry to clutter the thread, but in the interest of closing this issue, here is what I think we need:

1. respec citation for JSON Schema (probably specific draft version)
2. respec citation for JSON Pointer (probably specific draft version)

Until that is documented, I think it's safest to conclude that W3C cannot refer to either document informatively or normatively.

[plehegar]

Keep in mind that the list of considerations looks at how the reference gets used in the specification.

Looking at the https://www.w3.org/TR/vc-json-schema/#normative-references specification, it makes a reference to a living specification.

However, if you actually look at how it gets used, the specification requires some specific versions of JSON Schema for validation purposes. For forward compatibility purposes, I would think the normative reference to JSON-Schema seems appropriate.

[plehegar]

We don't and should not have a policy to reject references to the JSON Schema specification.

However, since JSON Schema is a living specification, the deeper you link into it, requires implementations to support specific features, or require validation, the more scrutiny/questions you'll get.

[jdesrosiers]

respec citation for JSON Pointer (probably specific draft version)

There's no need to reference a draft for JSON Pointer. It has a Standards Track RFC with IETF, RFC 6901.

[OR13]

Regarding references to JSON Schema "draft" versions, this is what we came up with:

    "JSON-SCHEMA-2020-12": {
      href: "https://json-schema.org/draft/2020-12/release-notes.html",
      title: "JSON Schema 2020-12 Release Notes",
      publisher: "OpenJS Foundation"
    },
    "JSON-SCHEMA-2019-09": {
      href: "https://json-schema.org/draft/2019-09/release-notes.html",
      title: "JSON Schema 2019-09 Release Notes",
      publisher: "OpenJS Foundation"
    },
    "JSON-SCHEMA-DRAFT-7": {
      href: "https://json-schema.org/draft-07/json-schema-release-notes.html",
      title: "JSON Schema Draft-07 Release Notes",
      publisher: "OpenJS Foundation"
    },

[decentralgabe]

As @OR13 points out we aim to reference the release notes from the OpenJS Foundation as a stable normative reference for each version See: https://github.com/w3c/vc-json-schema/pull/170

[jdesrosiers]

Every JSON Schema spec release has a stable URI. What's the benefit of referencing the release notes rather than the spec itself?

• 2020-12 - https://tools.ietf.org/html/draft-bhutton-json-schema-validation-01
• 2019-09 - https://tools.ietf.org/html/draft-handrews-json-schema-validation-02
• draft-07 - https://tools.ietf.org/html/draft-handrews-json-schema-validation-01

JSON Schema is a living specification

This is not correct. Maybe that has led you to believe that there aren't stable references to past releases?

[OR13]

My understanding from @Relequestual was that IETF was not the place to reference for JSON Schema draft versions, and that there are currently no plans to standardize JSON Schema at IETF.

[jdesrosiers]

We're not planning to standardize with IETF, but I don't see why you couldn't reference the drafts we've already published with them. We're going a different direction going forward, but those past drafts will always be hosted by IETF.

[mnot]

@jdesrosiers that's not true -- Internet-Drafts explicitly state:

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

Until relatively recently, Internet-Drafts were removed from the IETF Web site(s) once they expired. That changed, but there is no guarantee that they will persist.

[jdesrosiers]

there is no guarantee that they will persist.

That's good to know! We've been linking to past drafts from our website forever and never had a problem. The oldest having expired in 2010. We'll have to look into hosting copies ourselves.

And, yes, we are aware that we haven't used I-Ds the way they're intended. It's a big reason we're moving away from using them. Our circumstances don't align well with that process. Our past releases are in I-D format, but they really aren't true I-Ds because we haven't used them as such.

Perhaps the solution is for JSON Schema to self host the old drafts and projects like this one can reference those. We can't change that they are in I-D format. What's done is done and it's all we have. But, linking to them from a JSON Schema domain rather than an IETF domain is probably the best we can do to signal that they don't follow IETF norms. I know it's not perfect, but there has to be some way for it to be referenced because it's all we have.

Sorry for the trouble our misuse of I-Ds has caused. We are working on fixing that going forward.

[plehegar]

JSON Schema is a living specification

This is not correct. Maybe that has led you to believe that there aren't stable references to past releases?

Whether it's a LS or not isn't relevant for the VC/JSON-Schema spec in my mind.

Nevertheless, I appreciate the correction. The reason why I wrote that was because you wrote: [[ Instead of continuing to release a new immutable and incompatible version of JSON Schema with each release, our next release will be a long-lived version that is stable, but evolving. [...] That vision of a stable yet continuously evolving spec doesn't fit well with the IETF process. ]]

[jdesrosiers]

Yes, we're moving in that direction, but we aren't there yet. Anything released in the past doesn't apply to that vision and those past releases are what this discussion is talking about. I only mentioned this in case people thought that it applied to the releases being discussed here. When the time comes to reference a future release, things will be different.

I'll quickly add some clarification to what to expect from JSON Schema in the future, but keep in mind that none of this is set in stone yet. We're emulating ECMAScript's process, so you can generally expect JSON Schema to evolve in a similar way to JavaScript. When ECMAScript 2024 comes out it's not a new version of JavaScript, it's still the same old JavaScript, just with a few new features added. JSON Schema will work the same way rather than the old way of each release being a distinct and potentially incompatible version from the last.

Like ECMAScript, we intend to publish an maintain a full spec snapshot for each release. It won't be necessary to reference the release notes if you need to reference a specific feature set of JSON Schema. There will be, for example, a JSON Schema 2024 spec that you can reference directly.

[OR13]

I'd like to close off this thread with a simple example:

"heres how to cite JSON Schema, and specific versions of JSON Schema at the W3C"... then "mark issue as closed".

I still don't know how to do this.

[Relequestual]

I'd like to avoid marking this as closed for now. Thanks.

[plehegar]

@OR13, as a reminder, context matters here. Depending on how deep one uses JSON Schema will affect how to reference it. I simply suggest to make a proposal for your specific context.

[mnot]

Since this thread is still active :) a few points:

• Martin Thompson has floated a proposal to end expiry of Internet-Drafts. If that gains traction, citing the IETF URLs should be stable (although you still technically need to cite them as 'work in progress').
• The pushback from the IETF JSON community above was a long time ago now, and I suspect things may have changed, especially since the HTTPAPI community is now established there. And, even if the JSON greybeards are unexcited about it, that doesn't necessarily stop it -- especially if there's an active community that wants to bring work to the IETF.
• That said, JSON Schema has now been in development for several years and is widely used. Standards activity in the IETF or the W3C is likely to break things and cause disruption, because change control will pass to the standards body. That makes me think that the right thing to do might be to publish the JSON Schema community's preferred version (whenever it's ready) on the Independent Stream -- you get an RFC that other people (including the IETF and W3C) can cite normatively, but don't pass change control to the IETF. Then you can consider standardising JSON Schema 2.0 (if you like and where you like).

Happy to help / answer questions if there's interest in that path.

[Relequestual]

@mnot This sounds like something we should consider for sure. I think we would need to pick apart our ADR on decoupling from the IETF, and discuss if we can align on our expectation of travel. (Related comment from @awwright that suggests it would be possible.)

I think the points you make above are enough to be considered sufficient additional information to revisit the decision record results.

[jdesrosiers]

It's great to hear that removing expiration of I-Ds is being considered! Is there any chance this change could be retroactive and apply to previous versions of JSON Schema?

The Independent Stream is certainly a better choice for us in that it keeps change control with us, but I don't think it changes the core incompatibility we have with IETF. The normal process for RFCs is that you work through as many drafts as are necessary and then finish with an RFC. An RFC is expected to be done. Updates are expected to happen occasionally because real life is messy and the world changes, but planned regular updates are not an expected part of the normal process. Our expectation is that JSON Schema will continue to evolve for some time still. We are working toward moving to a model similar to ECMAScript where we introduce compatible changes on a yearly cadence. I've never heard of an RFC that worked that way. I expect we'd be told that if we need to be making updates that often, we're not ready for RFC and we should do more drafts. Meanwhile we're perpetually in the awkward position of developing something as a draft that's being used widely in productions systems. Please let me know if I'm misunderstanding anything here and you think there's a way such a process would fit within IETF.

[mnot]

To be clear, the Independent Stream isn't part of the IETF -- it's a separate user of the RFC Series. For that matter, Internet-Drafts are not monopolised by the IETF either; they're used by indivduals and other organisations (like the IRTF).

Regarding RFCs being "done", we don't consider that the case for HTTP (where four separate RFCs have defined HTTP/1.1 successively - 2068, 2616, 7230, and 9112). It's not the case for other successful protocols either, and in general there are many in the IETF who see the merits of continuous maintenance / "living specs".

You should be aware that the Independent Stream Editor decides what is included on the stream, so you should talk to him about what the process for publishing there would look like, to make sure you're comfortable. Generally, he won't meddle in the details of the document -- he just wants to make sure that what's on the stream is of reasonable quality.

[Relequestual]

To be clear, the Independent Stream isn't part of the IETF -- it's a separate user of the RFC Series. For that matter, Internet-Drafts are not monopolised by the IETF either; they're used by indivduals and other organisations (like the IRTF).

Well, that sounds interesting. What are the practical on the ground differences as a result? Could you like us to maybe a few RFCs from the Independent Stream? I did a little reading and it looks like the RFC could only be published as "Informational" or "Experimental". Is that correct?

Regarding RFCs being "done", we don't consider that the case for HTTP (where four separate RFCs have defined HTTP/1.1 successively - 2068, 2616, 7230, and 9112). It's not the case for other successful protocols either, and in general there are many in the IETF who see the merits of continuous maintenance / "living specs".

Very good and useful to know. Thank you. The pace of publication for those you mentioned may be slower than what we currently have planned, but that may be fine.

You should be aware that the Independent Stream Editor decides what is included on the stream, so you should talk to him about what the process for publishing there would look like, to make sure you're comfortable. Generally, he won't meddle in the details of the document -- he just wants to make sure that what's on the stream is of reasonable quality.

Sounds promising. Maybe our planned release schedule is something that could work with these type of submissions. It looks like the preferred contact method is to email rfc-ise@rfc-editor.org, correct?