Defining HTTP Headers in hyperschema

[davidjamesstone]

Hi there.

I'd like to validate HTTP headers with hyperschema, I can't see as it's part of the spec or any discussion around it.

Does anyone know if it's possible or if it's been brought up before ?

Many thanks.

[handrews]

@davidjamesstone not currently possible but I've been thinking of filing an issue on it. I'm not quite sure how it should fit in with hyper-schema.

For instance, while the HTML spec mentions HTTP headers it makes a point to indicate that it is using HTTP as an example and notes:

The HTTP headers are equivalent to fields in other protocols that have the same basic meaning. For example, the HTTP authentication headers are equivalent to the authentication aspects of the FTP protocol.

So even when we are thinking of JSON Hyper-Schema in terms of network APIs, we need to not be too HTTP-specific.

JSON Hyper-Schema is defining a media type. Unlike Swagger/OpenAPI/RAML/Blueprint, it is not a service definition that attempts to perfectly describe all aspects of an HTTP API. Something like (for instance) authentication/authorization is important for APIs but really is orthogonal to the choice of media type.

What specific headers do you have in mind and how do you see it fitting with JSON Schema?

[handrews]

@davidjamesstone I say "not currently possible" but "there is no specific way to address this" is more accurate. You can apply JSON Schema to anything if you want- if you want to pull your headers and their values out into a JSON document and validate it against a schema, nothing is stopping you from doing so. There's just no standard way to attach that to validation/link generation of message bodies.

[awwright]

@handrews identified the issue if you're looking to vary how you make HTTP requests in a hypermedia system described by JSON Schema. For that, I would just say, JSON Schema is designed for RESTful systems in general - without naming protocol-specific conventions.

Are you looking to perform testing of an HTTP server? It sounds like you might be asking for how to write a JSON vocabulary for representing HTTP headers in JSON.

Can you describe your particular use case a bit?

[davidjamesstone]

There's just no standard way to attach that to validation/link generation of message bodies.

I guess that's what I'm looking for.

I've been toying around looking into building a little web framework on top of hapi that's driven from a service definition file and validated using ajv. I've got something working based off swagger but I originally used JSON hypermedia.

Hapi supports validating against payload, params, query, headers and response. This dovetails quite nicely into what swagger does with it's parameters and in keys.

hyperschema is able to define all these element above apart from the HTTP Headers.

I had in mind using it for authorization tokens and such. It sounds like I may have confused the intent of hyperschema? There seems a lot of cross over between a service definition like swagger and what hypermedia can support.

Having the ability to describe what headers should look like in a link definition would complete the picture for my use case.

Thanks

[handrews]

This is my personal opinion, but in my view current service definition systems are attempting to do too much. They combine things that aren't sufficiently well-specified in media types, and that people just plain misunderstand (like standard HTTP semantics, which make a lot of common service definition features unnecessary if you follow them properly), with things that actually need to be specified.

JSON (specifically plain application/json) is not set up to be a hypermedia-enabled media type. So application/schema+json aims to cover that gap. An application/json document plus an application/schema+json document gives you a fully hypermedia-enabled system (plus structural validation).

Somewhere something should be able to describe things like how headers get used, how auth is handled, etc. That thing probably looks more like a service definition, but a very minimal one as it would not describe the resource representations or links. It would just cover those things that are not covered by media types, so it would work equally well for JSON (with or without JSON Schema) or XML representations.

AFAIK, no such thing exists, but I haven't really gone digging. On my todo list...

[davidjamesstone]

Thanks for that, I took me a while to digest but I think I get the distinction now. I think you mean that the purpose of JSON Schema hypermedia is to only describe the resource and it's relations, not how it's obtained. And service definitions are conflating the two?

If so, that makes total sense to me and it's obvious the more I think about it.

TBH I'm still tempted to (ab)use JSON hyperschema as a service definition of sorts and see how it works out in place of swagger. I'm quite taken by the idea of using a single spec (JSON Schema and hyperschema) to describe all entites and api endpoints and use this metadata to build systems including database, api validation, docs generation, ui form generation and auto-tests. I'm thinking this would make a nice modern base for a CRM or ERP product or something.

Anyway - thanks again for clearing that up. I hope I've understood you correctly.

[awwright]

@davidjamesstone That sounds right. JSON Schema tells you how to interpret JSON documents, it doesn't tell you how to interact with a remote server.

HTTP already specifies how you can interact with it. For example, authentication is specified in the WWW-Authenticate header. While you could assert "The server will always send this WWW-Authenticate header" in a service document, you'd end up losing many of the benefits of loose coupling (a property of RESTful protocols like HTTP).

[handrews]

I think you mean that the purpose of JSON Schema hypermedia is to only describe the resource and it's relations, not how it's obtained. And service definitions are conflating the two?

Yes, exactly! Conciseness is not one of my strengths :-)

I am also trying to work out an as-much-JSON Schema-as-possible approach. It will at some point get open sourced (assuming all goes well), and it will be fascinating to compare notes once I can!

UI Form generation: See #67 (filed by the maintainer of Angular Schema Form who is also splitting a framework-neutral library out and I believe is in touch with the person doing react-jsonschema-form as well.

Documentation: I'll actually be involved in getting doca more in conformance with JSON Schema and particularly hyper-schema. It's a little off in spots right now but we'll be working on that soon. No schedule, no promises, but it's definitely a thing that will happen.

API validation: I haven't looked at hapi yet.

[davidjamesstone]

This has really helped me and I've learned a lot. Thankyou both.

I'll keep in touch with how things go.

BTW hapi is great, you should check it out. They have their own take in validation with Joi which you've probably come across.

I read they initially considered JSON Schema but this was some years ago and they decided to go ahead with their own custom approach at the time.

Slotting in AJV was pretty easy though and it feels much better (and is faster in some basic tests I have done).

[handrews]

@davidjamesstone great! Do keep us posted (separately, I encourage you to file any concerns or suggestions with doca over at the cloudflare repo since we're going to be working on it soonish).

Slotting in AJV was pretty easy though and it feels much better (and is faster in some basic tests I have done).

I'm sure @epoberezkin will be happy to hear that :-D I know he put a lot of work in on the performance side.

I have seen Joi but if I understand correctly it is too JavaScript-centric for general validation use. I haven't seen hapi before- server side stuff I've been working with in the last few years has been in Python or Go so I'm behind the times on JavaScript server frameworks :-)

[handrews]

@davidjamesstone since this is more about usage and best practices, I filed issue json-schema-org/json-schema-org.github.io#63 on documenting this sort of thing on our web site. If that looks like it would address your questions, could you go ahead and close this issue? Or if you think there's anything that needs to be done with the spec please comment on that here.

[davidjamesstone]

Great @handrews - I look forward to reading the first pass.

Thanks again for the explanation, I can see now HTTP Headers do not belong in here and I'll close this issue.

However, like you say:

Somewhere something should be able to describe things like how headers get used, how auth is handled, etc. That thing probably looks more like a service definition, but a very minimal one as it would not describe the resource representations or links. It would just cover those things that are not covered by media types, so it would work equally well for JSON (with or without JSON Schema) or XML representations.

This is also now on my todo list too!

[Relequestual]

@davidjamesstone Just seen this issue! All seems to make sense. I'm personally still unsure how I'll be documenting my own API, however I have had RAML in my sights for almost as long as JSON Schema. RAML allows the use of JSON Schema, and also allows you to define the sorts of elements you'd expect when documenting an API, like headers!

Recently I've been looking into making some hyperschemas for the API I'm working on, but having been reminded that it still doesn't cover some aspects (like headers) made me remember RAML. Personally I have issues with Swagger and how their "based on JSON Schema but modified" approach raises interoperability issues, while RAML just goes "yeah, JSON Schema or some other definitions, either is fine".

[handrews]

@Relequestual I need to take another look at RAML then... :-)

[Relequestual]

(I know I'm breaking my no discussion rule, but as this issue is already closed...)

My problem with the api-console (https://github.com/mulesoft/api-console) is that it doesn't show you the description and constraint information for each field within the JSON Schema. It just plonks the Schema there for you to see. Doca however shows each field. I may just go with Doca and write manually about the headers, but the problem then is the CURL requests will be wrong.

[handrews]

See cloudflare/doca#5 for follow-up on headers.

And yeah, doca is trying to be more user-friendly than reading raw JSON Schema, and we will continue to improve the UX there.

[handrews]

@davidjamesstone After a fair amount of discussion we decided that it makes sense to model protocol metadata (a.k.a. headers) that can be used with links. draft-07 hyper-schema (which should be published next week) adds targetHints (for response header values) and headerSchema (for request header validation schemas) to the link description objects.

This went back and forth a few times, but hopefully draft-07 makes it clear why we eventually decided that headers fit, while a number of other API concepts do not.

[davidjamesstone]

That's great @handrews. I've been going through draft 7 and it's looking great. You are prolific and you can see the effort you and others have put in. Congratulations!

I'd like to contribute more but I find internet drafts quite dry and hard to interpret. I'm much more of an examples and try/fail kind of person when it comes to learning. If there's any way I could help though I'd be glad to.

I'm resurrecting the work I started (with hapi+schema) last year and see if I can get some examples published.

Thanks

[handrews]

but I find internet drafts quite dry and hard to interpret

I'm pretty sure they're required to be that way ;-)

Glad to hear you're working in this area again! We definitely need a more friendly tutorial to go alongside of the specification. For now, I'm focusing on writing a test suite so that if/when someone implements draft-07 Hyper-Schema, they can determined whether they implemented it correctly or not.

[Relequestual]

@davidjamesstone If you'd like to help an already well used resource update their tutorials, https://spacetelescope.github.io/understanding-json-schema/ needs updating from draft-4. I don't know if they have issues logged or not, but their work is really helpful to the community. It was super helpful to me when I first started investigating JSON Schema!

[xpepermint]

Congratulations guys! This looks awesome.

@handrews how far are you with the test suite? I'm not sure I use HTTP headers as expected. A validator would be nice.