First Implimentation of a GOC API

[samperd]

Hey Folks,

In the spirit of Agile I am wondering if there is an appetite to propose a first implementation of a unified GOC API end point. Let me know if this is off topic or already being explored.

The proposal would be to work with SSC (Shared Services Canada) to stand up an API proxy. This proxy would provide a single end point that people can connect to (eg http://data.gc.ca/api).

This nice part is that SSC would be the host and have to relatively little development.

Employing an API proxy already deployed in large Open Stack environments we can implement something quite quickly. I propose that we can explore Open Repose http://www.openrepose.org/ which is part of the Open Stack () and already heavily used, developed and supported.

The idea is that each API owner would need to configure the reverse proxy to consume their own in house API.

This would be a quick win because we can standardize (and evolve the standard) by controlling the requests coming in and leave it to the departments to configure their own API.

This should be a fairly simple activity with a quick(ish) turn around time. (3 months for first launch and 3 API's?).

If this activity already belongs with someone within GOC please point me in the right direction. This would be fun to help advise.

Cheers

[LaurentGoderre]

I'm wondering what the usability of one massive API would be.

[chrismajewski]

I don't know of any work to that end, as of yet. It's been greatly talked about in terms of a single API for canada.ca, canada.ca/api-ipa to be specific, with every other API only sharing endpoints which is next to impossible itself unless we had something similar to the US Umbrella platform the Web Interoperability Working Group (WIWG) has been lusting for.

http://api.data.gov/about/ Centralized delivery for cost savings ( bandwidth, reduction in servers, etc... ), metrics, metering, and so much more.

This would be an interesting discussion with a whiteboard prior to CODE http://www.canadianopendataexperience.com/

[samperd]

Great Question.

let me try to hit a few points.

1. I am picturing something like what google does (https://developers.google.com/apis-explorer/#p/). Google is a web property with different API hooks and perhaps different functionality. however many of the most common API's have a similar structure. If you notice, many of their properties are dropping the subdomains in favour of URL hierarchy. This would do the same for GOC API's
2. To support massive adoption of all the goodies GOC already offers, bringing them (API's) under a single domain may increase their use. It also increases the probability of uptake.
3. Many [GOC] API's may have a common vocabulary (eg name, geo-location, date, measurement) that can be explained at a high level. Then specialized API's (eg Weather, Earthquakes, mapping services and postal code lookups) would be able to further extend the meta API. Through a single domain and leveraging URL's. This might be explained using the concept of separate Atom Workspaces (http://bitworking.org/projects/atom/rfc5023.html#workspaces)
4. All documentation for the meta API would be at one location.
5. Combining data from within the API may be easier and perhaps fewer API calls.
6. Many departments have multiple API's using different specs, the proxy could be the encoder/decoder for these API's. Especially for API's created before standards came into play or that use obscur standards.
7. More consolidation will likely occur into SSC in the future (as with WEB and E-MAIL) and departments may loose internal support for their specialized API's. This is common story through my experience and indicative of future trends. Having a single API front end may help in this consolidation.
8. Some smaller departments may not have the capacity to support their own API but may have great data. these departments can help build out the meta API in a centralized manner.
9. The web is about platforms and perhaps we need to think about the GOC as a platform (http://www.youtube.com/watch?v=dYB8xokkWjg)
10. The reverse proxy idea would still leverage a federated approach for organizations to build and maintain their own API's. This federation would be transparent to the user and developer.

Ten is a nice round number to stop at (smile)

The idea is not to centralize, control and conquer, but rather provide a single access point for Canadians to developer against the GOC platform.

Cheers

[samperd]

@chrismajewski "with every other API only sharing endpoints which is next to impossible itself unless we had something similar to the US Umbrella platform". Is this a statement "for" or "against" a central reverse proxy?

When I worked at NRCAN we did a similar approach using a [homebrew] proxy to pull together some geo-locations services. It turned out not too badly and did not break existing users using the separate end points (documentation: http://geogratis.gc.ca/loc/en/).

Three services (GeoNames, NTS and Postal Code) were brought together under a single endpoint.

I am just proposing to extend this model to the GOC family.

Cheers

[chrismajewski]

From the WIWG a central reverse proxy is the goal slightly out of reach ATM.

We've been talking about it in a larger scope than the proxy and generally, in the light of consolidation, the idea of being a step back on APIs where they are all over the place doesn't seem to fit well at the 30-40k foot view. From the implementation end your starting to stack different requirements into a pipe, the political pressure at that pipe is extraordinary. Too much pressure at that end and you risk losing more in adoption than you hoped for.

It'd be a great place to be, well worth a face to face with the right people to chew on. If this can be sorted for CODE it's a win across the board. A central hub, however, has cost and manpower associated with it, we'd need to find that.

A few things to throw at you.

Consider API stacking as per Roy Fielding's work. Consider internal vs external APIs and their relationships. Consider overlaping APIs. Consider a central GitHub approach for all public interfacing.

I guess we can start talking about this, we are examining the possibility of centralizing documentation and public human comment/bug/etc to APIs in a single GitHub org. We'd like to say code as well but there are logistical nightmares down that path as it reaches too deep into departments, their security and data requirements ( international agreements, privacy, legally bound release dates, etc... )

Each API owner gets to control that as they would but overall the public has a central contact point, we start centralizing the documentation and conversations ( internal and external for both ).

[samperd]

Fair enough @chrismajewski ,

I just thought an implementation can help solicit requirements for a standard to quickly iterate so we get to the standard end of things quicker.

Where would be the best place to discuss with interested parties further?

Cheers

[chrismajewski]

I have the Earthquake API running to the standards as they were about 2 months ago. I'll have it running to standards come CODE or when the standard is formalised. I'm more than happy to keep that up to date as the running model.

I should be at 270 Albert starting mid-December, this puts me in the same space as @dyomides and Ashley. Can we sort these details when I'm there?

If we can't land it before the holidays it would be shortly after.

[chrismajewski]

The Earthquakes API is being rebuilt to meet the standard as it sits.

I'm hoping to have that ready in a couple weeks barring issues. It will not be registered or released till we are reasonably content with the draft standard so it can be used as a guinea pig for testing.

[LaurentGoderre]

Can you send me the WIP files?

[chrismajewski]

I have a call to Laurent to look over some XML, I have a second API to convert up to the standard as it is now.

Next week I have something we can all poke at.

[chrismajewski]

Didn't happen "next week" but with a bit more time to consider it it's not as crazy as it seems.

To note for discussions with the Metadata Experts Working Group ( MEWG ) on API metadata. It's an identified need for APIs and the MEWGs expertise to comment on.

[chrismajewski]

Hopefully the MWS can find a buyer for application migration using this as a starting point.

This will happen through the TBS::CIOB::MWS Config team if so.