zdne commented 11 years ago

Direct support for hypermedia and modeling actions & relations.

rurounijones commented 11 years ago

I am surprised that this is marked for 1B, imho there are other, more real-world use issues that would be better implemented in 1B and leave Hypermedia until 1C

zdne commented 11 years ago

I believe Hypermedia elegantly solves many real-world issues such as client decoupling, versioning (along with content negotiation), discoverability to name a few. It also directly contributes to better API design, maintainability and scalability.

The idea is to bake its support directly into API Blueprint so you do not even have to think whether you are hypermedia or not. For example this could automatically provide the location HTTP header where applicable or construct the link HTTP header etc. In cooperation with #10 this could go even further utilizing media types such as Collection+JSON, Siren or HAL.

Idealistically, should you choose to, I would like you to think about resources, relations and actions not HTTP verbs, URIs and such, but that is definitely even beyond the 1C..

mikekelly commented 11 years ago

Has anyone sketched out what this might look like?

zdne commented 11 years ago

@mikekelly, this is how I project it at this very moment:

In API Blueprint (Markdown), using some semantic assumptions:

Define a resource with a name, id or an URI
Describe the resource model media type (state, attributes)
Describe relations between the model and other resource models
Specify a transaction example with a hypermedia affordance provider media type content negotiation

The API Blueprint parser (or its harness) will take the care of injecting the actual model media type and relations (possibly even actions) into the hypermedia affordance provider media type.

Using part of @lukestokes foxycart API something along these lines (again, just sketching here):


# Resource User [User]

+ Model (application/json)

{
  "first_name": "Qing",
  "last_name": "woldu",
  "email": "test481598556@example.com",
  "phone": "12345678",
  "affiliate_id": 0,
  "is_programmer": false,
  "is_front_end_developer": false,
  "is_designer": false,
  "is_merchant": false,
  "date_created": "2012-02-29T13:55:09-0800",
  "date_modified": "2013-07-11T15:28:56-0700"
}
```

Relations
- self = [User][] ... This User
- stores = [Stores][] ... Stores for This User

Retrieve a User [GET]

Request (Accept: application/json)
Response 200 (application/json)

[User.json][]
Request (Accept: application/hal+json)
Response 200 (application/hal+json)

[User.hal+json][]

Resource Stores [Stores]

Model (application/json)

...



The API Blueprint parser would take care of injecting the _user `application/json` model_ into the respective response media type in its resulting AST (and [AST media types](https://github.com/apiaryio/snowcrash/wiki/API-Blueprint-AST-Media-Types#json-serialization)) e.g. do just a verbatim copy for `[User.json][]` or build the `application/hal+json` representation in the case  of `[User.hal+json][]` using the model and relations.

Note I would also love to have Siren on board alongside with HAL in the first round. Considering other provides later. Also this concept can be extended to auto generate the **Link HTTP header**.

This is in a nutshell how I think about this issue at the moment. There are still few pieces of puzzle missing but that is alright for now. Please let me know what you do think and whether it can work at all.

Thank you! 

---

Note for @kevinswiber : This is the topic we didn't have the chance to discuss at #apicraft.

lukestokes commented 11 years ago

I'll let smarter people than myself comment on the approach, but let me just say I'm really excited about seeing progress in this direction! Our sandbox supports both HAL and Siren, so I hope it can be a useful "real world" test bed for trying some of this stuff out. If I can help in a specific way, please let me know. Here are some of the keys I can think of as it relates to the needs client developer:

Educate and enforce the importance of REST principles (think in terms of media types, response codes, caching concerns, resources, link relationships, etc and less in terms of remote procedure calls to URLs).
Use visualizations in the documentation that help them navigate relationships to find the resources of the API they are interested in. As a real world example, we have users who only want to integrate with the API so they can generate and manipulate coupons. Building a documentation interface that helps them navigate to that relationship will be great.

One thing I was also considering is that not every API has to (or will be) a true REST API. If there's a clear way to distinguish between JSON RPC APIs and REST Hypermedia APIs, that would be fantastic.

Great work, Z. I'm really looking forward to where this goes.

zdne commented 11 years ago

@lukestokes I do agree in all points but the very first one – you are very qualified, if not the best, for commenting on this issue!

Regarding the clients – as we have discussed with @cainus its the clients where we need to improve the situation next. Not by telling them what they should do but pointing real life benefits as well as offer the best tools to consume our APIs. Now imaging sticking an hypermedia-enabled API Blueprint into a client SDK to configure it for your API and provide realtime request checks etc... (OK now I am totally going both off topic & off the roof :)

mikekelly commented 11 years ago

I think the only reasonable way of achieving broader adoption is to ship APIs where clients will code against them without even knowing what hypermedia is. This means we need media types that people can treat as plain JSON and they are still just as easy to work with and understand.

If I was you, I would pick the media types you initially adopt here carefully. If you offer something that is too complicated, and not intuitive enough - you are going to undermine your larger objective.

But I would say that, obviously. :trollface:

zdne commented 11 years ago

Mike, while I do agree with this line:

ship APIs where clients will code against them without even knowing what hypermedia is

However I do believe a broader adoption would be achieved through providing API SDKs and/or tighter framework client integration rather than finding the "holy grail" of media types. I treat a media type just as a mean to transfer the resource representation and, in some cases, hyperlink relations. Idealistically a client should be interested in (and exposed only to) actual resource data and the relations, not in the technical means of the communication itself.

In another words as a client you should only need to know your data are and where to get the next.

This is exactly what I would like to achieve with API Blueprint. In the first step to sweep the technical means of a hypermedia API under a nice and tidy Markdown carpet, so you do not have to really think HAL, Siren or anything like this – you just focus on your data and relations. Second do the same on the consumer's end – using an API Blueprint on client to hide the HTTP / transfer technicalities and provide directly the data and relations.

What do you think?

lukestokes commented 11 years ago

It's funny, prior to the RESTfest talk by @bsletten , I would have been worried about hiding things behind an SDK or client library. But now, if REST is truly used under the hood, I can see that the format and the datastore don't matter as much, as long as information can be treated as addressable resources which tell you what can be done next. His keynote video isn't up on vimeo yet, or I'd link to it.

kevinswiber commented 11 years ago

@zdne

However I do believe a broader adoption would be achieved through providing API SDKs and/or tighter framework client integration rather than finding the "holy grail" of media types. I treat a media type just as a mean to transfer the resource representation and, in some cases, hyperlink relations. Idealistically a client should be interested in (and exposed only to) actual resource data and the relations, not in the technical means of the communication itself.

:+1:

We care more about the media types, because that's what we do. Most people just want to get the job done.

The difficult thing when supporting multiple hypermedia types on the same server is transforming capabilities between media types. If one media type only supports links while another supports controls, do you whittle down to the least common denominator? If so, why have additional media types? Do you provide enough information for capabilities of ALL media types supported? If so, do you just discard that information when rendering media types that don't support it?

These aren't very easy problems to solve in some cases. I'm curious what route @lukestokes took with FoxyCart.

kevinswiber commented 11 years ago

Note for @kevinswiber : This is the topic we didn't have the chance to discuss at #apicraft.

Hey @zdne, sorry we didn't get a chance to discuss.

Here's how I've been looking at Siren APIs, in terms of documentation.

Things to define:

Classes
Link relations (for entities and links)
Actions

Entities of a particular class may have certain properties, sub-entities, actions, and links. (Note: An entity can have multiple classes.) Defining this is important in documentation. I use class values to route responses to appropriate handlers on the client side. This allows the client to respond to the actual message from the server instead of setting an expectation at request time. For some, this is a different way of thinking, but it's an extremely powerful, reactive pattern.

Everything flows from the entity class documentation. Sub-entities have link relations that should be defined. Actions should be defined, and so should expected input fields. Hidden input fields, of course, should not be defined. The Siren spec instructs clients to resubmit hidden fields.

As far as functional documentation (docs that are able to submit requests), with Siren, this has to be dynamic. One of Siren's benefits is communicating available actions at request-time, based on resource state. Potential state transitions can change, and so, too, should the active docs.

lukestokes commented 11 years ago

These aren't very easy problems to solve in some cases. I'm curious what route @lukestokes took with FoxyCart.

We are just at the lowest common denominator right now. I wanted to include actions in our Siren support (and still plan to in the future), but we didn't want to go there until we finalized more important link relationships and property naming conventions.

I also want to support Collection, though I'd probably have to do an extension of it because we want to include properties of the collection such as total_items, returned_items, limit, and offset. In the future I may look into XHTML as well.

The formatting of the output, imo, isn't nearly as hard as properly designing the API in the first place. As long as the API has a good format layer in place, it's just a matter of moving data around. We did run into some funkiness with how things are done in JSON vs. XML, but for the most part, I think we came up with good approaches that wouldn't appear as compromises. Our hope in supporting multiple formats is that we can be a "real world" API for people to start testing their hypermedia tools against. More tools = more adoption.

Feel free to play around with it. I'd love any feedback you have: https://api.foxycart.com/docs

Any way, as it relates to this discussion, maybe it would make sense to include @mamund H-Factor ideas when thinking about the documentation system. Just a thought.

dilipkrish commented 11 years ago

I'm just dipping my toes into this hypermedia ocean. The featured apiary apis look great, hadn't discovered it until recently! Apiary looks really powerful! @lukestokes the foxy cart api looks like it already supports hypermedia and consequently serendipitous api navigation discovery!

Anywho, as a background, I've been trying to socialize RESTful services and take it to the next level (of maturity) @ The Container Store. As a start, we've been using swagger to "document" our internal API's. Thats a great start, and it seems like a very similar approach to apiary. I've been thinking about how to make the simple API's hypermedia aware, and secondly how to make them discoverable once it is. Like @mikekelly said,

I think the only reasonable way of achieving broader adoption is to ship APIs where clients will code against them without even knowing what hypermedia is.

My 2 cents; considerations when it comes to hyper media, documentation and restful services

The generation of the documentation cannot be deliberate, as in, distinct from the act of designing the API/service itself. So either the service can allow the generation of service information or vice versa. That way each side does not have the risk of being out of date. So the workflow is really important for adoption.
Secondly, the whole notion of "documentation" falls flat with respect to a hypermedia API. By definition for a hypermedia API you should only need to document the entry points (as proposed by @zdne) the rest should be inferred as a result of navigation and discovery. Otherwise, IMHO it smells of the RPC style documentation that we got with SOAP, WSDL and its band of brothers.

Glad to help take this discussion forward.

mikekelly commented 11 years ago

@kevinswiber

We care more about the media types, because that's what we do. Most people just want to get the job done.

I'm in the latter camp for sure. Hal happened because I wrote a gist to show what I was building, then bunch of other people got interested so made a little website, then people started asking for a spec...

I don't care that much about media types - I care about helping people build effective, usable APIs.

There are some media types which I, as someone who's been around this space for a bit, feel are basically over-complicated and solving problems nobody's even sure they have - which is fine in and of itself, but when it comes to projects like this is worth highlighting as a risk for the reason you've outlined here:

The difficult thing when supporting multiple hypermedia types on the same server is transforming capabilities between media types. If one media type only supports links while another supports controls, do you whittle down to the least common denominator? If so, why have additional media types? Do you provide enough information for capabilities of ALL media types supported? If so, do you just discard that information when rendering media types that don't support it?

Don't get me wrong, I think pushing things further is worthwhile, but I think picking the low-hanging fruit first and then figuring out where to go next is a more reasonable approach. It sounds like this is exactly what @lukestokes has done with the foxycart API - I'll be interested to see where they go next.

zdne commented 11 years ago

@mikekelly

I care about helping people build effective, usable APIs.

Excellent point Mike! This is my aspiration with API Blueprint.

Again looking at FoxyCart HAL Browser: Basically my idea is to describe what is in the left pane in a Markdown syntax and let API Blueprint parser generate what is on the right. Without any prior knowledge of HAL or HTTP / REST.

Just tell me about your data and the relations. Simple.

zdne commented 11 years ago

@kevinswiber

Thanks for your feedback and tips on Siren, much appreciated!

Indeed if we decide to go for both the approach of @lukestokes with the smallest common denominator is the way to go.

Now, it seems to me the (choice of) (hyper)media type is really more important when you have one end open. Should there ever be something like an API Blueprint driven client consuming an API designed with API Blueprint then the whole (hyper)media type would be really just a 'technicality' under the hood.

mikekelly commented 11 years ago

Yes that sounds like a really good idea. fwiw, when I spiked the haltalk demo - this is what the link relation documentation ended up looking like:

http://haltalk.herokuapp.com/rels/signup

zdne commented 11 years ago

Yay! This looks almost like an API Blueprint:

signup

Create an account [POST]

Request

Headers

Content-Type: application/json

Body

{
  "username": "fred",
  "password": "pwnme",
  "real_name": "Fred Wilson"
}

Response 201
- Headers
```
Location: -----
```

Very helpful. Thanks for sharing!

kevinswiber commented 11 years ago

@mikekelly

I'm in the latter camp for sure. Hal happened because I wrote a gist to show what I was building, then bunch of other people got interested so made a little website, then people started asking for a spec...

You're in both camps. Everyone has a stake in their work, and hopefully, the belief that it's helpful.

There are some media types which I, as someone who's been around this space for a bit, feel are basically over-complicated and solving problems nobody's even sure they have

Siren isn't an academic project. It came out of real work, as well. It was solving my problems. Now I'm helping others use it to solve their problems. Siren does have capabilities HAL doesn't. It's okay if you don't need them. Other people do. This conversation between us is a broken record that is being played on 3 different mailing lists. I'd like to show @zdne some respect and not pollute this thread with nerd battle. I'm happy to take this to the Siren list if you'd like to play this song together once more.

kevinswiber commented 11 years ago

http://haltalk.herokuapp.com/rels/signup

@mikekelly @zdne

So a question I run into from time to time asks if something like this is actually over-specified. The usage of HTTP headers and status codes is already well-defined. You could really just say "Follow HTTP" for a lot of this.

Also, there are some things missing that I'd care about as a client developer. What about validation? When might there be an error condition? Are usernames unique?

I'm also starting to favor leading with description rather than HTTP, but that's mostly a formatting thing. I'd prefer to highlight "Creates an account" over "POST," because that's the bit client developers are really going to care about.

mikekelly commented 11 years ago

I'd like to show @zdne some respect and not pollute this thread with nerd battle

great, me too! It's not a personal affront, I'm just offering advice that I think makes sense for this project at this point in time.

Also, there are some things missing that I'd care about as a client developer. What about validation? When might there be an error condition? Are usernames unique?

Yep - there are many more things this description document could contain, haltalk is a really basic demo - it's not a finished product.

The main point of the demo is to show that the various transitions of an API can be broken down into, documented, afforded, and discovered via link relations. It's a no-nonsense strategy for making an API discoverable and moving the attention away from URL patterns.

kevinswiber commented 11 years ago

Now, it seems to me the (choice of) (hyper)media type is really more important when you have one end open. Should there ever be something like an API Blueprint driven client consuming an API designed with API Blueprint then the whole (hyper)media type would be really just a 'technicality' under the hood.

I want to say yes, because this sounds interesting, and anything that sounds interesting is worth doing, in my opinion. Innovation is a big, messy pile of ingredients. Sometimes you just need to throw 'em in the cauldron and see what comes out.

Media types offer different capabilities, and that's where it gets tricky. No one cares too much about links vs _links. When you're looking at consuming an API, what you care about is features offered by the provider. I think we all agree on this.

The question then becomes, as an API provider and potentially someone who will create clients for various platforms, "Which media types help support the features I'm looking to implement?" This is where making a mistake really kind of stinks. You can start off with something minimal and try to bolt-on later, but eventually, the Frankenstein monster emerges... or you intentionally cripple the capabilities of your API. (Those words sound harsh, but it's not always so terrible.)

So the argument for starting with a media type that has the most capabilities is there. Hydra[1] is another media type I've taken a liking to lately. I need to examine it more closely, but I would submit Hydra as a candidate for another one of these "capabilities-rich" media types.

Of course... you could always come up with whatever format you think would work for you in this situation and birth something like application/blueprint+json. :)

You should also take a look at ALPS[2] if you haven't already. It's aiming to add supporting application semantics to APIs.

I feel like CCing some people since I'm talking about their work, and surely, they know more about their work than I do. /cc @lanthaler @mamund

Now... light that fire and get that cauldron boiling. ;)

[1] http://www.markus-lanthaler.com/hydra/spec/latest/core/ [2] http://alps.io

lukestokes commented 11 years ago

When I started thinking about docs, I figured people building their businesses on our platform will want more information rather than less. I've only got some twig templates in a twitter bootstrap theme, but I thought about having a short paragraph summary of the link relationship along with the affordances (we're a CRUD-heavy api so one relationship does multiple things via various HTTP methods, something OPTIONS can also help discover). Additionally, I want to show the property with detailed descriptions, types and constraints. We also want to show some example outputs in the various formats we support. It's a work in progress, but here are some examples:

http://api.foxycart.com/rels/stores http://api.foxycart.com/rels/transaction http://api.foxycart.com/rels/create_client

When it comes to embedded resources, I also include stuff from /rels/store inside of /rels/stores. Something to think about as users may work with a collection in a slightly different manner than a single resource.

I don't think there will ever be a "one-size-fits-all" solution to everyone's needs, but at the very least, focusing on link relationships instead of URLs will be a big step forward, IMO. As long as things are extensible so I could add in my own details, that would be great.

Really excited about where this is going, guys. Thanks for encouraging me with your efforts. :)

kevinswiber commented 11 years ago

http://api.foxycart.com/rels/stores http://api.foxycart.com/rels/transaction http://api.foxycart.com/rels/create_client

Hey, I think this looks pretty good!

Capturing all the elements you have outlined there in a more abstract way would get something closer to the hypermedia version of WADL.

@lukestokes Thanks for actually implementing a real hypermedia API with public documentation. It's helpful for the whole community and hopefully helpful for FoxyCart, as well.

lanthaler commented 11 years ago

So the argument for starting with a media type that has the most capabilities is there. Hydra[1] is another media type I've taken a liking to lately. I need to examine it more closely, but I would submit Hydra as a candidate for another one of these "capabilities-rich" media types.

Thanks for the nice words and cc'ing me Kevin. Unfortunately, I think I don't have much to add at this moment. I just wanted to say that if there are any questions regarding Hydra I'm here to answer them :-)

mamund commented 11 years ago

I know this thread is getting quite long; good stuff here. I wanted to add one more thing to the mix when considering how to go about documenting a "Hypermedia API"

I've recently gotten into the habit of documenting the actions, not the objects. I have a (very) rough example posted in another repo[1]. Right now this approach takes a couple steps (not ideal):

doc the media type
doc the domain
provide examples (combine the media type and the domain)

I think that is too unwieldy and can be greatly improved. However, my point here is to pitch documenting the hypermedia controls, not the resources. Its a very diff approach, but something I think can work well. It allows servers to decide their own URIs, their own representation details (what data and which controls appear in responses) and still gives client devs enough to write a working client (that recognizes both the data and the controls when they show up in a response).

Anyway, its just another approach to consider.

Cheers.

[1] https://github.com/apiacademy/class-scheduling/blob/master/docs/problem-domain.asciidoc#actions

mikekelly commented 11 years ago

Which media types help support the features I'm looking to implement?

More features in the media type does not translate to more features of a service. Compromising on media type "features" and instead, folding that complexity into something else (e.g. link rel documentation) does not make a service "inferior". It is a design decision.

more can be less.

mikekelly commented 11 years ago

Defending complexity by offering more "features" is a well proven sales technique, though - I'll grant you that.

kevinswiber commented 11 years ago

@mikekelly I acknowledge your opinions. I have a feeling individual stubbornness on both our parts will prevent compromise. You admit HAL is intentionally minimalist yet claim anything more complex is a waste. For some reason, you think HTML is the only other media type that gets to play in the game. You haven't recognized any positive aspects regarding any other hypermedia type. That is not very constructive to the conversation. I get it. You want everyone using HAL exclusively. That won't happen, in my opinion, because of its minimalist design. I can program with GOTO all day long, but you know what? Sometimes I like IF-ELSE structures, too. You are a very loud minority, and your opinions, though acknowledged, are losing value.

kevinswiber commented 11 years ago

@mamund This approach is very interesting. Are the actions for a link ID always static or could they change at runtime?

mikekelly commented 11 years ago

You admit HAL is intentionally minimalist yet claim anything more complex is a waste.

I would be less inclined to claim that if the complexity in question is presented along with a demonstration of the additional (evolvability-related) behaviours it affords, rather than hand-waving and talking about "having more features".

You are a very loud minority, and your opinions, though acknowledged, are losing value.

This strikes me as a bit unnecessary, but I'm sorry you feel that way.

mamund commented 11 years ago

@lukestokes yes. ATM, the control identifiers are static and part of the "domain knowledge" required by the client app (just like data elements (fname, lname, etc.).

There is some generic language about the appearance of "unknown" control identifiers somewhere else in the text. I think it reads something like: "For any control identifier that is unknown to the client, the client SHOULD ignore that control. However, clients MAY present the control for human interaction and, in this case SHOULD treat the control action as unsafe and non-idempotent."

This describes a kind of "worst case" scenario for unknown hypermedia controls. Currently, my client apps ignore the unknown controls (e.g. bots don't activate them and human-facing apps don't render them).

On Thu, Oct 3, 2013 at 4:15 PM, Kevin Swiber notifications@github.comwrote:

@mamund https://github.com/mamund This approach is very interesting. Are the actions for a link ID always static or could they change at runtime?

— Reply to this email directly or view it on GitHubhttps://github.com/apiaryio/api-blueprint/issues/13#issuecomment-25654165 .

zdne commented 11 years ago

@mikekelly @kevinswiber

Gentlemen,

I am really happy to have both of you in this discussion but please allow me to turn your precious attention to the question that is slowly building in me ever since I have started thinking about the Hypermedia support in API Blueprint:

Do you think that another level of abstraction above the REST (and Hypermedia) can actually work out? Didn't Hypermedia failed (in the terms of adoption) just because its already too abstract for many?

I feel like both @mamund's ALPS and ours API Blueprint are (by definition) abstract from HTTP and REST and now we basically want to sweep the Hypermedia under the carpet. Which from one point of view be looked at as a simplifying of things; yet on the other side it means increased complexity through abstraction?

In another words, can we abstract from REST enough to provide a simpler way to define, build and consume our services (that would be under the hood Hypermedia, HTTP and TCP/IP driven)? Should we even try?

kevinswiber commented 11 years ago

@zdne

Yes, and yes!

For most developers, utilizing remote services is not about REST or hypermedia. The protocols, media types, and architectural stylings are all means to an end. If the focus is on providing a good experience for developers to provide and consume remote services, I think this is very doable. The underlying benefits of REST should remain, but again, these are details under the hood.

Not to scare anyone away, but this essentially mimics the "Generate WSDL"/"Import WSDL" ease of use on top of protocols that are infinitely more scalable and hopefully with "generate"/"import" features that don't require extreme maintenance to avoid breakage (evolvability, decoupled implementation stuff).

mamund commented 11 years ago

@zdne https://github.com/zdne

FWIW, my comments here (and my descriptions of ALPS) are not meant to "sweep the Hypermedia under the carpet." In fact, my approach is to do the opposite. Make the documentation centered around the hypermedia controls; not the resources.

I understand this approach is another abstraction. I know this is a risk. Frankly, I'm not too worried ATM since I've not seen anyone using this and failing miserably yet (since its not widely used at all ;). I want to see how this works, whether it is accepted, whether it actually solves a real problem. Then we can talk about it's value.

I got no evidence yet, so I continue on ;)

On Thu, Oct 3, 2013 at 4:47 PM, Z notifications@github.com wrote:

@mikekelly https://github.com/mikekelly @kevinswiberhttps://github.com/kevinswiber

Gentlemen,

I am really happy to have both of you in this discussion but please allow me to turn your precious attention to the question that is slowly building in me ever since I have started thinking about the Hypermedia support in API Blueprint:

Do you think that another level of abstraction above the REST (and Hypermedia) can actually work out? Didn't Hypermedia failed (in the terms of adoption) just because its already too abstract for many?

I feel like both @mamund https://github.com/mamund's ALPS and ours API Blueprint are (by definition) abstract from HTTP and REST and now we basically want to sweep the Hypermedia under the carpet. Which from one point of view be looked at as a simplifying of things; yet on the other side it means increased complexity through abstraction?

In another words, can we abstract from REST enough to provide a simpler way to define, build and consume our services (that would be under the hood Hypermedia, HTTP and TCP/IP driven)? Should we even try?

— Reply to this email directly or view it on GitHubhttps://github.com/apiaryio/api-blueprint/issues/13#issuecomment-25656657 .

lukestokes commented 11 years ago

@mamund

Make the documentation centered around the hypermedia controls; not the resources.

Very interesting (and thank you for linking to an example). Question about your example... does it assume a flat structure? That, to me, is the missing piece... if, as a client developer, I want to do something, I have to understand how the relationships between resources work in order to get to the right place, right? We're back to WWBD (What Would Browser Do) and thinking about the UI/UX aspect of how the API will be used. Or maybe the client can figure this out on their own as long as it understands the domain and the controls?

I do think any step away from just thinking in URLs (which, I guess would be thinking purely in resources) and step toward thinking in terms of controls / actions / link relationships would be really helpful.

zdne commented 11 years ago

@mamund

Pardon, I have expressed myself wrong – I want to sweep the Hypermedia technicalities (headers, media-types etc.) under the carpet so we can actually focus on and leverage the hypermedia qualities.

Indeed I want to promote the Hypermedia way of thinking; just maybe without explicitly saying it. I still need to think about your controls-first approach. It is intriguing indeed. It suddenly feels odd living in the model & relations world.

As you are definitely ahead by far in this field I will try to enjoy the luxury and learn from your example.

zdne commented 11 years ago

@kevinswiber

Thanks Kevin! Much appreciated. I am not biased with WSDL so that definitely won't scare me away! (Albeit I strive to learn from the old mistakes)

mamund commented 11 years ago

@lukestokes

yes, my messages are almost always flat. it is the easiest agreement. my experience so far (esp. w/ enterprise-level work) is that it is much easier to get agreement on a shared vocabulary (fname, lname, etc.) than an object model (person, product, etc.). for this reason, my message designs reflect this approach.

however, this is a design decision, nothing more. if someone wants to design messages that ship around objects/graphs, etc. that's fine w/ me ;)

On Thu, Oct 3, 2013 at 5:11 PM, lukestokes notifications@github.com wrote:

@mamund https://github.com/mamund

Make the documentation centered around the hypermedia controls; not the resources.

Very interesting (and thank you for linking to an example). Question about your example... does it assume a flat structure? That, to me, is the missing piece... if, as a client developer, I want to do something, I have to understand how the relationships between resources work in order to get to the right place, right? We're back to WWBD (What Would Browser Do) and thinking about the UI/UX aspect of how the API will be used. Or maybe the client can figure this out on their own as long as it understands the domain and the controls?

I do think any step away from just thinking in URLs (which, I guess would be thinking purely in resources) and step toward thinking in terms of controls / actions / link relationships would be really helpful.

— Reply to this email directly or view it on GitHubhttps://github.com/apiaryio/api-blueprint/issues/13#issuecomment-25658445 .

mamund commented 11 years ago

@zdne

yeah, i think we're on the same page. np.

On Thu, Oct 3, 2013 at 5:13 PM, Z notifications@github.com wrote:

@mamund https://github.com/mamund

Pardon, I have expressed myself wrong – I want to sweep the Hypermedia technicalities\ (headers, media-types etc.) under the carpet so we can actually focus on and leverage the hypermedia qualities.

Indeed I want to promote the Hypermedia way of thinking; just maybe without explicitly saying it. I still need to think about your controls-first approach. It is intriguing indeed. It suddenly feels odd living in the model & relations world.

As you are definitely ahead by far in this field I will try to enjoy the luxury and learn from your example.

— Reply to this email directly or view it on GitHubhttps://github.com/apiaryio/api-blueprint/issues/13#issuecomment-25658611 .

mikekelly commented 11 years ago

I feel like both @mamund's ALPS and ours API Blueprint are (by definition) abstract from HTTP and REST

Hm.. but , iirc, the blueprint response description examples are actually expressed in terms of HTTP?

now we basically want to sweep the Hypermedia under the carpet. Which from one point of view be looked at as a simplifying of things; yet on the other side it means increased complexity through abstraction?

I think you should focus on building stuff that helps people make sense of the actual messages that are going over the wire, because this is the stuff that people will poke around with, discover, debug, etc. I haven't seen many people asking for more abstraction in APIs recently, but I have seen them asking for more accurate documentation and less breakage of their client code.

zzen commented 11 years ago

For me +1 for @mikekelly. For me the "lessons learned" from SOAP/WSDL is "don't use abstraction as an excuse to make things complicated under the hood". Because at the end of the day, when sh*t hits the fan, under the hood is where you'll end up, poking around with some sort of debugger.

But things could work a bit more more out-of-the-box. Think (optional) Rails-style convention-over-configuration.

kevinswiber commented 11 years ago

Abstraction does not drive complexity; it's the other way around. HTTP is complex. Hypermedia can be perceived as complex. The solution isn't to dilute functionality. The solution is to properly utilize a design that serves as both usable and functional. Complexity never actually dilutes, anyway; it just gets shuffled around. Tucking it behind unspecified formats for link relation documentation does not make it disappear. You hit the problem later than sooner, but you still hit it nonetheless.

Anyway... I'll be adding some prototyping in this space to my backlog soon. It's too interesting to not start coding. :)

Sent from my iPhone

On Oct 3, 2013, at 7:13 PM, "Jakub Nešetřil" notifications@github.com wrote:

For me +1 for @mikekelly https://github.com/mikekelly. For me the "lessons learned" from SOAP/WSDL is "don't use abstraction as an excuse to make things complicated under the hood". Because at the end of the day, when sh*t hits the fan, under the hood is where you'll end up, poking around with some sort of debugger.

But things could work a bit more more out-of-the-box. Think (optional) Rails-style convention-over-configuration.

— Reply to this email directly or view it on GitHubhttps://github.com/apiaryio/api-blueprint/issues/13#issuecomment-25666138 .

mikekelly commented 11 years ago

Tucking it behind unspecified formats for link relation documentation does not make it disappear. You hit the problem later than sooner, but you still hit it nonetheless

What do you mean by "unspecified formats"? Do you mean an HTML page of documentation written in plain English? HTML is already specified and most people are fine reading English (or some other language). Having said that it would probably be useful to standardise on a template for how link relation documentation should look, just so people have less to think about. I think that is something this project could establish nicely and it sounds like @zdne is already working towards it, anyway.

Nobody said link rel documentation will make the complexity of a service disappear, but it does draw a very clear line in the sand where the parts of the API that are intended for machines belong (the API message) and where the parts of the API intended for developers belong (the link relation documentation). Adding more controls into the message just muddies where that line is drawn, and it doesn't really buy you anything wrt machines.

If you need forms because your API is actually intended as a GUI, then serve your controls as HTML (e.g. disqus comments). HTML is everywhere and it's very well understood - there's really no good reason to try and reinvent its functionality in JSON.

kevinswiber commented 11 years ago

@mikekelly Again... automation in dynamic workflows is not possible without an affordance that communicates available state transitions at response time. With Siren, I'm evaluating moving away from HTML-defined types for input fields. There are pros, but there are also cons. GUI benefits just come free. The benefit during M2M is runtime validation. This means a field type can change on the fly from 'text' to 'email' and automatically reap data cleanliness benefits. A client can deny input before it even submits the request. There are other ways to do this, and that's what I'm planning to evaluate.

Putting this information in the message has the added benefit of ease in developer exploration. Just surfing the API, a developer can learn what is needed to continue down the path toward a goal... just like machines, but able to make smarter decisions or change directions once in motion.

By "unspecified formats," I meant there's typically no link relation documentation that is automatically consumed by the client for the purpose of dynamic runtime client configuration. This is what would be required for the workflow use case that doesn't include this information inline. This sort of thing could certainly exist some day. Without something like this, I haven't seen a good design for this use case that doesn't involve inline actions of sorts.

These are the benefits I need, and it's easier for both machines and developers if the information is included in the message body itself (no extra requests required).

I often code in state machines, and I like to avoid dummy requests. If the server communicates available transitions, along with metadata describing the parameters for executing those transitions, this reduces network chatter and makes "where to go from here" crystal clear. That's beneficial for a number of reasons.

mikekelly commented 11 years ago

automation in dynamic workflows is not possible without an affordance that communicates available state transitions at response time.

Couldn't you use links for each of the transitions? Why would that not be sufficient?

With Siren, I'm evaluating moving away from HTML-defined types for input fields. There are pros, but there are also cons. GUI benefits just come free. The benefit during M2M is runtime validation. This means a field type can change on the fly from 'text' to 'email' and automatically reap data cleanliness benefits. A client can deny input before it even submits the request.

Well, in that case you end up with less visibility over which clients you just broke. You could just handle this server side and respond with a validation error message. The basic behaviour is equivalent (the client will break) but approach is better for the client because:

Their lives are easier because they don't need to implement pre-validation.
You have more visibility into the effect of your change, since you can log invalid requests that failed server side - and track how quickly those clients get fixed.

This is a really good example of where adding complexity and "features" actually adds no value and arguably does the opposite.

Putting this information in the message has the added benefit of ease in developer exploration. Just surfing the API, a developer can learn what is needed to continue down the path toward a goal... just like machines, but able to make smarter decisions or change directions once in motion.

And yet link relation documentation is able to convey the exact same information except with the benefit of being as free-form as necessary to get the point across. Witness the hal browser. All without adding noise to the messages at runtime thanks to the indirection of link relation URLs.

By "unspecified formats," I meant there's typically no link relation documentation that is automatically consumed by the client for the purpose of dynamic runtime client configuration.

A link relation URL is free to serve as many different content types as it likes via the usual HTTP content negotiation stuff - you're not limited to text/html. I don't know of many people who do this though - perhaps because they don't feel its worth it in practice? I think I remember a couple of HAL projects a couple of years ago that were also serving schema docs from the link rel URLs. No idea how that turned out though...

This is what would be required for the workflow use case that doesn't include this information inline. This sort of thing could certainly exist some day. Without something like this, I haven't seen a good design for this use case that doesn't involve inline actions of sorts.

I really think it would help if you could produce a concrete example of one of these mysterious "dynamic workflows" and ideally include examples of the sorts of dynamic changes you think clients should be able to adapt to and how these changes manifest themselves in the messages.

These are the benefits I need, and it's easier for both machines and developers if the information is included in the message body itself (no extra requests required).

Well, these are the affordances you think you need, because you believe it improves ease-of-use enough to be worth the additional complexity.

There is always a trade off to be made.

I often code in state machines, and I like to avoid dummy requests. If the server communicates available transitions, along with metadata describing the parameters for executing those transitions, this reduces network chatter and makes "where to go from here" crystal clear. That's beneficial for a number of reasons.

I don't know what you mean by "dummy requests" so its impossible to comment on this.

Cheers, M

fosdev commented 11 years ago

@zdne

So, walking into this discussion principally, if blueprint is an api design tool, imo it should support API design from an abstract to a concrete standpoint in a media-type and protocol independent fashion. That is the media-type and protocol are implementation details that follow after the the basic definition of the resource and its available affordances (ideally with a state machine in mind).

Basically, that means:

I define the data semantics of my resource (what the metadata and/or attributes mean).
I define the affordances associated with my resource and any requisite parameters or attributes that should be supplied with those affordances (relations, link or form).
I define the statemachine associated with the resource (albeit this is more for internal documentation than external, but effectively this is important because it highlights the sets of affordances that one might experience and the conditions that must be satisfied for the affordance to be present in a response).

At this point if I have done my job correctly I have a RESTful resource defined in its entirety and if I want to generate a machine-readable profile, say like ALPS, I can. I have as @mamund suggests, effectively defined the application domain profile associated with the resource. This to me is fundamental to API design and very important to support M2m applications.
I then define the media-types that my api supports and can supply sample representations.
I then define protocol specific implemenation details for the affordances (uris and uniform interface methods, headers, status codes, etc). However, from a hypermedia API standpoint

As you and I have discussed, I think the design tool should support user defined metadata in the tree that is part of the blueprint, but not part of the documentation. So, introduce into the blueprint tree a metadata tag in my description below. The metadata tag could be used anywhere in the tree and can be used to tracking media-type specific attributes as well once you get into the issue of supporting different types.

Past that, I am not necessary advocating the naming below, per se, but just using examples that I think explain the ideas. Note: I leave any statemachine specific stuff out of the example for now.

Format: Hypermedia

# Profile Contact
This is a contact.

## Properties
  + name - The contact name
  + phone - The contact phone

## Affordances
  + self
    + metadata
        + type - safe
  + update - Updates the contact
    + metadata
      + type - idempotent
    + Parameters 
      + ...
    + Body
      + name
      + phone

## Media-types
Contact representations in different media-types.

### application/json
### application/hal+json

## HTTP

### contact/{uuid}
  + self
    + GET 
      + Headers ...
      + Response 200

  + update
    + PUT
      + Headers ...
      + Response 201

The API documentation associated with this might look like:

Profile Contact

This is a contact.

Properties

name - The contact name
phone - The contact phone

Relations

self
update - Updates the contact
- Parameters
- ...
- Attributes
- name
- phone

Media-types

Contact representations in different media-types.

application/json

  {
    "name": "bob",
    "phone": "555-1212"
  }

application/hal+json

  {
    "_links": {
      "self": { "href": "..."},
      "update": { "href": "..." },
    },
    "name": "bob",
    "phone": "555-1212"
  }

HTTP

self

GET
- Headers ...
- Response 200

update

PUT
- Headers ...
- Response 201

Note the absense of the metatdata in the documentation. I think with a little creativity, much of the existing tree could be used and some of your markdown conventions aliased as necessary. It may be that you could use the format Metatdata at the to add aliases, etc. E.g. Media-type for Model (that latter which feels very RPC to me).

Also, we can put serializers into this that can generate the sample media types and even ALPS profiles (plug for @mamund).

zdne commented 11 years ago

Personally I am really aligned with what @fosdev suggests.

However I would like to take some shortcuts initially. For example to build on HTTP protocol. Being protocol-agnostic is still a long-term goal however I would sacrifice it for clarity and simplicity. At least in the beginnings.

Also I am not sure whether the concept of tagging is necessarily related to this Issue's topic.

As we have agreed with @fosdev we will try to explore this approach in-depth. Recently, I have published a new API Blueprint Tutorial that is "built" on top of an imaginary API (blueprint). Since this imaginary API described in the blueprint is RESTful and yet close to an existing real-world I would like to take use it as an archetype API for this research.

steveklabnik commented 10 years ago

I'm just adding my :+1: here, am interested to see how it turns out.

glennblock commented 10 years ago

OK, weighing in extremely late (kudos to @zzen for point this out).

First this is a great discussion and even more importantly it has brought in many of the people who really care about Hypermedia who have been giving thought leadership on the topic, so thank you! I also would to like see things move forward here as I've been on my own quest for Hypermedia and quest for educating on Hypermedia. (http://shop.oreilly.com/product/0636920026617.do)

Here are a few of my own thoughts that I'll throw into the mix:

Having multiple options is good. The space is still young and we're learning. I am a big believer in the crowd sourced natural selection that happens around OSS efforts. It will happen. If that means we end up with multiple solutions, than great, if we all naturally hone in on one, then that's also fine.
Please at least support a way to express Hypermedia. I think THIS is critical. Folks are starting to get on board both big and small with bringing Hypermedia into their designs. Not because they attend the church of Fielding, but because they see value. Don't block them. Don't make them say, "We like Blueprint (or some other doc format) but we can't use it because our API is Hypermedia driven". Depending on the success of such format, they will either lose by having to give it up in favor of their API or lose in giving up Hypermedia.
Let's keep our eyes on the prize. Let's work together to move the bar, to make it easier for people to build hypermedia systems and not get caught up on which way is better than the other.

I am a big fan of the discussion here and I think that efforts like this can play a big part in advancing the state of the art so to speak.

apiaryio / api-blueprint

Hypermedia #13

Retrieve a User [GET]

Resource Stores [Stores]

signup

Create an account [POST]

Profile Contact

Properties

Relations

Media-types

application/json

application/hal+json

HTTP

self

update