Can we alter the API for more consistency?

[dgpratt]

Certain aspects of the API as it is now defined are bothering me. For example, the API to retrieve a single organization is "GET /organization/{organization}" and to get a single fleet within an organization is "GET /{organization}/fleet". Why does the 'organization' prefix or namespace not appear in subsequent APIs? I would prefer a URL scheme where an identifier is always preceded by its...um...container, for lack of a better word. For example:

POST /organizations GET /organizations/{organization}

POST /organizations/{organization}/fleets GET /organizations/{organization}/fleets/{fleet}

POST /organizations/{organization}/fleets/{fleet}/vehicles GET /organizations/{organization}/fleets/{fleet}/vehicles/{vehicle}

POST /organizations/{organization}/fleets/{fleet}/vehicles/{vehicle}/locations GET /organizations/{organization}/fleets/{fleet}/vehicles/{vehicle}/locations/{?start,end}

I think the above is more consistent within itself as well as compared to other Web API schemes I have encountered.

[dgpratt]

On a related note, I wonder if I misunderstand the role of the 'organization' entity in this schema. Does not the organization represent the legal entity that is consuming this service? If so, I'm not sure I see the sense in having an API to create an organization. It seems like that's something that would be handled 'out of band' when an organization signs up for the service. By analogy, I doubt there is an API for provisioning an Amazon Web Services or Windows Azure account.

If, in fact, my understanding is correct, I think the API as defined could be collapsed down a bit, like so:

POST /{organization}/fleets GET /{organization}/fleets/{fleet}

POST /{organization}/fleets/{fleet}/vehicles GET /{organization}/fleets/{fleet}/vehicles/{vehicle}

POST /{organization}/fleets/{fleet}/vehicles/{vehicle}/locations GET /{organization}/fleets/{fleet}/vehicles/{vehicle}/locations/{?start,end}

[garrettwilkin]

If you notice the /{organization} notation, which is distinct from simply /organization in the path.

Let's assume that the organization ID would 011, for instance,

then you would have a call very similar to the ones you suggest.

The difference is that the sections of the URL wrapped in braces (i.e. { } represent IDs for individual entities.

{organization} represents an ID for an organization. {vehicle} represents an ID for a vehicle. And so on.

Also, why do we need to leave things to be done out of band? Is that advantageous?

Garrett Wilkin voice: (207) 370-1862 profile: http://about.me/garrettwilkin

On Fri, Oct 11, 2013 at 3:46 PM, dpratt71 notifications@github.com wrote:

On a related note, I wonder if I misunderstand the role of the 'organization' entity in this schema. Does not the organization represent the legal entity that is consuming this service? If so, I'm not sure I see the sense in having an API to create an organization. It seems like that's something that would be handled 'out of band' when an organization signs up for the service. By analogy, I doubt there is an API for provisioning an Amazon Web Services or Windows Azure account.

If, in fact, my understanding is correct, I think the API as defined could be collapsed down a bit, like so:

POST /{organization}/fleets GET /{organization}/fleets/{fleet}

POST /{organization}/fleets/{fleet}/vehicles GET /{organization}/fleets/{fleet}/vehicles/{vehicle}

POST /{organization}/fleets/{fleet}/vehicles/{vehicle}/locations GET /{organization}/fleets/{fleet}/vehicles/{vehicle}/locations/{?start,end}

— Reply to this email directly or view it on GitHubhttps://github.com/mainehackerclub/open_vehicle_tracker/issues/29#issuecomment-26166458 .

[Derrick-]

I kind of liked the simplicity of using just the names in the API in that it was compressed, but only to the extent that everything is distinctly routable.

The way it is now gives you URLS such as: http://openvehicletracker/api/Derrick/Trucks Would list all the trucks. I really like the way this url looks. And it makes sense to be until we get to the locations one, at which point it loses consistency and begins to require a named parameter ‘location’.

I like using the plurals, as I wasn’t comfortable with POST /{organization}/{fleet}/vehicle returning a list.

Your suggestion does make everything more explicit, but maybe we can compress it a little without losing that?

Just as food for thought how does the following seem?

POST /{organization}/fleets GET /{organization}/fleets/{?fleet}

POST /{organization}/{fleet}/vehicles GET /{organization}/{fleet}/vehicles/{?vehicle}

POST /{organization}/{fleet}/{vehicle}/locations GET /{organization}/{fleet}/{vehicle}/locations/{?start}

I think you may still want the following even if only for Administrators: GET /organizations POST /organizations

From: dpratt71 [mailto:notifications@github.com] Sent: Friday, October 11, 2013 3:46 PM To: mainehackerclub/open_vehicle_tracker Subject: Re: [open_vehicle_tracker] Can we alter the API for more consistency? (#29)

On a related note, I wonder if I misunderstand the role of the 'organization' entity in this schema. Does not the organization represent the legal entity that is consuming this service? If so, I'm not sure I see the sense in having an API to create an organization. It seems like that's something that would be handled 'out of band' when an organization signs up for the service. By analogy, I doubt there is an API for provisioning an Amazon Web Services or Windows Azure account.

If, in fact, my understanding is correct, I think the API as defined could be collapsed down a bit, like so:

POST /{organization}/fleets GET /{organization}/fleets/{fleet}

POST /{organization}/fleets/{fleet}/vehicles GET /{organization}/fleets/{fleet}/vehicles/{vehicle}

POST /{organization}/fleets/{fleet}/vehicles/{vehicle}/locations GET /{organization}/fleets/{fleet}/vehicles/{vehicle}/locations/{?start,end}

— Reply to this email directly or view it on GitHubhttps://github.com/mainehackerclub/open_vehicle_tracker/issues/29#issuecomment-26166458.

[dgpratt]

Garret, I'm not sure we understand each other, so I'm going to try to make an example. Suppose there is an organization 'DanCorp' that has a fleet 'Plows' and at least one vehicle in that fleet, 'P1'. By my understanding of the current API spec, to GET the organization entity, the URL would be:

/organization/DanCorp

And to get the fleet entity, it would be:

/DanCorp/fleet/Plows

And finally, to get the single vehicle, it would be:

/DanCorp/Plows/vehicle/P1

And under the scheme I have proposed, the URLs would be:

/DanCorp
/DanCorp/fleets/Plows
/DanCorp/fleets/Plows/vehicles/P1

Respectively.

Now to Derrick's point about compressing the URLs, is there a good technical argument for trying to compress them? I can think of a couple counter-arguments. For one thing, the explicitness makes the API more amenable to future expansion. It seems very possible to me that the day may come when we want to organize something else besides fleets directly under an organization. For another thing, I guess we can argue the point endlessly, but I really like the idea of 'tagging' each piece of variable data in the URL.

If we do want to shorten them as much as possible, then its not obvious to me why we would need to have any 'fixed' terms in the URLs at all. It seems we could simply do:

/DanCorp
/DanCorp/Plows
/DanCorp/Plows/P1

I'll respond to the 'out of band' question in a follow-up comment.

[dgpratt]

As far as whether it makes sense to be able to create an organization entity within the API, it depends greatly on some assumptions about how this API once implemented will actually be used.

By the way, most or all of what I'm about to say assumes that the 'organization' in our scheme is the legal entity that will be using the web service we are defining and that legal entity is not (necessarily) the same entity that is hosting the service. If that assumption is incorrect, most of what follows will not make sense.

Who do we envision will be hosting this web service? And will it be acceptable to them that any anonymous internet entity can come along and create one or a thousand 'organization' entities and start pushing data into them. Doesn't seem right to me. Rather, it makes more sense to me that there will be a human-oriented sign-on process, which will create the organization entity. Maybe money will be exchanged or maybe not, but at least there'll be a barrier to abuse.

Also there is the important matter of how calls into this API will be authorized. It seems that most web API providers today provide a 'secret' as part of the sign on process (which also implies that there is a sign on process). I'd guess that most providers rely on SSL to keep the secret a secret, but there are alternatives, which is good because it seems SSL will not be practical from the kind of devices we envision being used. One such alternative is HMAC (apparently Amazon Web Services uses it). But HMAC also uses a shared secret scheme. How does one securely transfer the shared secret? A separate secure sign-on process!

[garrettwilkin]

I don't think that URL length matters technically. I'm more concerned with implementing than hashing out a perfect spec. We can always iterate to v2 after learning what we did wrong in v1. If you would like to change the spec, please do. We might even be able to share access to the apiary.ioservice.

I think its beneficial to allow as much as possible to be done in the API, and have very little that requires a human to intervene. The only part that would not be handled in the API would be the generation of an API key. The API key is used to authorize the creation of organizations and also the adding of further data under that organization.

The idea I had when designing the spec was a hosted service model, in which multiple organizations send in GPS time series data to the same hosting service.

The purpose of having the ability to create an organization via API is that the "admin" web interface use the API.

Garrett Wilkin voice: (207) 370-1862 profile: http://about.me/garrettwilkin

On Sat, Oct 12, 2013 at 10:22 AM, dpratt71 notifications@github.com wrote:

As far as whether it makes sense to be able to create an organization entity within the API, it depends greatly on some assumptions about how this API once implemented will actually be used.

By the way, most or all of what I'm about to say assumes that the 'organization' in our scheme is the legal entity that will be using the web service we are defining and that legal entity is not (necessarily) the same entity that is hosting the service. If that assumption is incorrect, most of what follows will not make sense.

Who do we envision will be hosting this web service? And will it be acceptable to them that any anonymous internet entity can come along and create one or a thousand 'organization' entities and start pushing data into them. Doesn't seem right to me. Rather, it makes more sense to me that there will be a human-oriented sign-on process, which will create the organization entity. Maybe money will be exchanged or maybe not, but at least there'll be a barrier to abuse.

Also there is the important matter of how calls into this API will be authorized. It seems that most web API providers today provide a 'secret' as part of the sign on process (which also implies that there is a sign on process). I'd guess that most providers rely on SSL to keep the secret a secret, but there are alternatives, which is good because it seems SSL will not be practical from the kind of devices we envision being used. One such alternative is HMAC (apparently Amazon Web Services uses it). But HMAC also uses a shared secret scheme. How does one securely transfer the shared secret? A separate secure sign-on process!

— Reply to this email directly or view it on GitHubhttps://github.com/mainehackerclub/open_vehicle_tracker/issues/29#issuecomment-26198304 .

[Derrick-]

If we do want to shorten them as much as possible, then its not obvious to me why we would need to have any 'fixed' terms in the URLs at all. It seems we could simply do: /DanCorp /DanCorp/Plows /DanCorp/Plows/P1 I considered this myself, but it’s not routable given that some of the params are optional.

[Derrick-]

As far as whether it makes sense to be able to create an organization entity within the API, it depends greatly on some assumptions about how this API once implemented will actually be used. I don’t see any necessity for restricting a user from creating multiple organizations within the protocol itself. I think the protocol should allow the ability; and that ability can be restricted by policy outside of the scope of protocol.

Any level of the api can be subject to security, authentication, and permissions.

[Derrick-]

Now to Derrick's point about compressing the URLs, is there a good technical argument for trying to compress them? I can think of a couple counter-arguments. For one thing, the explicitness makes the API more amenable to future expansion. It seems very possible to me that the day may come when we want to organize something else besides fleets directly under an organization. For another thing, I guess we can argue the point endlessly, but I really like the idea of 'tagging' each piece of variable data in the URL.

I like the explicitness of the URL's you proposed, however is it a good idea to add data unless that data provides some element of distinction? I.e., the static routing path information seems superfluous to me. Although I think the last named parameter for each becomes necessary in order to avoid ambiguity.

If this was a function call with named parameters I would see the readability benefit; but in this case It seems to me that it confuses the readability because the additional parameter may be interpreted by a reader that there were some sort of additional options provided by them, where there is not.

As opposed to naming optional parameters in a function call which allow you to rearrange the order of them etc... I think tagging is good if it implies options.

[cano64]

API URLs don't have to be nice, SEO or human readable, designing API URLs in the following format has advantage of universality and future expansion

GET key1/value1/key2/value2/..../keyN/valueN

about authentication arduino doesn't have enough processing power for SSL HMAC might be an option, but I'm not sure, we can also use a co processor for encryption, either dedicated solution or just another arduino chip to offload calculations from main processor. (this way you can turn it off if not used to conserve batteries)

[dgpratt]

Derrick, can you give an example of an ambiguous route if we completely omitted the fixed terms?

[Derrick-]

POST /{organization}/ GET /{organization}/{?fleet}

POST /{organization}/{fleet} GET /{organization}/{fleet}/{?vehicle}

POST /{organization}/{fleet}/{vehicle} GET /{organization}/{fleet}/{vehicle}/{?start}

Given the above: GET /{organization}/{?fleet} (List of fleets, or single fleet) and GET /{organization}/{fleet}/{?vehicle} (List of vehicles or single vehicle)

are not distinguishable because of the optional param. if I GET: /Derrick/Trucks You don't know if I'm asking for the single {fleet} item, or a list of vehicles in the {fleet}

[dgpratt]

Oh, I think I see. So in the existing scheme, if I do:

GET /DanCorp/fleet/Trucks

That means get me the single 'Trucks' fleet.

But if I do:

GET /DanCorp/Trucks/vehicles

That means get all the vehicles in the 'Trucks' fleet? I missed the "get all" part before. What if someone did something silly like make a fleet named "fleet". And then did:

GET /DanCorp/fleet/vehicles

How would that get interpreted?

[Derrick-]

What if someone did something silly like make a fleet named "fleet".

In my .Net MVC implementation I created a reserved word list to avoid this. see: https://github.com/OpenVehicleTracker/Servers_DotNet/blob/master/dotnet.openvehicletracker.org/Models/Entities/BaseNamedEntity.cs

 private static readonly string[] reservedNames = { "organization", "fleet", "vehicle", "location" }; 

In the original implementation, the only ambiguous route I discovered was with the name 'location', but I opted to make all the route param names reserved.

API URLs don't have to be nice, SEO or human readable, designing API URLs in the following format has advantage of universality and future expansion

I very much agree with this when the key/values are not dependent upon each other or optional, or if order is intended not to matter; however in this case there is an explicit requirement for organization to be specified if fleet is also specified, etc. so I think the requirement of "parent" parameters dictates order in this case, in the end giving a specific and unique Uri for each query

That is to say, with the key/value arrangement both of the following would be valid and equivalent: GET: /api/fleet/Trucks/organization/Derrick GET: /api/organization/Derrick/fleet/Trucks

I think it's better to enforce a expected Uri parameter order: primarily because I don't see any value-added benefit to supporting variability of the arrangement of the Uri params

[Derrick-]

Thinking more about this:

universality and future expansion Yeah: though it does provide that benefit; it also introduces a lot of invalid cases for example: /fleet/Truck/vehicle/Jeep is invalid because there's no organization Strict routing reduces case checking.

[dgpratt]

Derrick, I have to say that I really do not like the reserved name thing. It's a good example of the accidental complexity that tends to get introduced when you base stuff off of an inconsistent model.

Also, cano64 (if that's your real name), were you suggesting that the parameters should be reorderable? I didn't get that from your comments. I just assumed you were saying that any identifier should be preceded by its container.

[cano64]

yes, they can be. the whole thing will get transformed into an associative array and individual routers will just check if specific key exists in the array

[Derrick-]

Yeah I don't like the reserved name thing either, it shares the same un-feature that I was just complaining about.

I'm not sure if the reserved names are needed in the proposal I made though:

POST /{organization}/fleets
GET /{organization}/fleets/{?fleet}

POST /{organization}/{fleet}/vehicles
GET /{organization}/{fleet}/vehicles/{?vehicle}

POST /{organization}/{fleet}/{vehicle}/locations
GET /{organization}/{fleet}/{vehicle}/locations/{?start}

[garrettwilkin]

Also dont forget about using URL parameters in the API.

http://www.parsely.com/api/single/index.html#document-api_ref

Check out how the Parsel.ly API is specified. There are optional parameters, and those are included like this:

GET http://api.parsely.com/v2/posts?days=1&limit=10

In that case, days and limit affect the result set that is returned on the API call. But of course you can filter by record values, not just by time & quantity.

http://openvehicletracker.apiary.io/{organization}/{fleet}/vehicle/{?name}

This call would be:

GET http://openvehicletracker.apiary.io/AcmeCorp/Plows/vehicle/?name='Bubba'

to retreive the plow named bubba.

Another alternative:

GET http://openvehicletracker.apiary.io/vehicle/?name='Bubba'&Organization='AmceCorp'&fleet='Plows'

[dgpratt]

Well, we seem to be moving farther and farther away from a consensus, which wasn't my intention :)

I guess as with most things, whoever makes something useful first will likely get to set the standard :)

[Derrick-]

This is a great conversation. The different viewpoints have really got me thinking.

I tend to agree with Garrets suggestion that the place for unordered (and potentially optional) key/value pairs belong in the query string, and the path as the name suggests should be very strongly structured and not contain superfluous elements. Just as you would not make a directory structure in which a directory would be restricted by protocol to contain only a single element.

[Derrick-]

http://openvehicletracker.apiary.io/{organization}/{fleet}/vehicle/{?name} This call would be: GET http://openvehicletracker.apiary.io/AcmeCorp/Plows/vehicle/?name='Bubba'

I completely misinterpreted the api definition: I thought that http://openvehicletracker.apiary.io/{organization}/{fleet}/vehicle/{?name} meant http://openvehicletracker.apiary.io/Derrick/Trucks}/vehicle/Jeep not http://openvehicletracker.apiary.io/Derrick/Trucks}/vehicle/?name=Jeep

[dgpratt]

Since my other attempts to convince you all of my world view have not worked as I had hoped, I shall now attempt to "appeal to authority". I've done some research on popular REST APIs. Services where identifiers appear in URLs seem to fall into one of two categories: (1) what I call the "bucket" method -- each entity gets its own API, with no apparent relation to other entities (at least as far as the URL is concerned), and (2) the "hierarchy" method, where it works like the current OVT API.

In either case, all the APIs seem very consistent as far as always preceding identifiers with the container name.

Here's some URLs of what I looked at:

https://developers.google.com/+/api/ https://developers.google.com/bigquery/docs/reference/v2/ http://msdn.microsoft.com/en-us/library/windowsazure/ee460799.aspx https://dev.twitter.com/docs/api/1.1 http://developer.netflix.com/docs/Common_Tasks http://www.flickr.com/services/api/ http://www.twilio.com/docs/api/rest/transcription

[Derrick-]

Neat :) That's kind of what I was looking for in my comments on the API page itself: Is this some sort of standard pattern: It looks like it is.

What do you think about the Uri structure at this point? You said whoever implements it first gets to define it, but I'm of the position that You, Cano, Garret and I are doing that first implementation together right now :)