AgCan (MBB) Web Services perspective

[gnewton]

Hello,

This comment is as a result of a telephone call I had with Laurent several Fridays ago. He suggested I submit an issue containing what we discussed.

I am Glen Newton from Agriculture Canada. Here are our organizational repos: https://github.com/AAFC-MBB Here are my repos: https://github.com/gnewton

We both produce and consume web services, both internally and externally. Most of this is for scientific data sets, but also includes things like an internal LIMS (laboratory information management system). You can see some of these in our repos. We also have internal-only repos.

We have evolved the JSON (no XML) that we are producing. Right now we are working towards a standard that we hope could merge in to WET. It consists of:

• http://jsonapi.org/
• Most of https://google-styleguide.googlecode.com/svn/trunk/jsoncstyleguide.xml (with some things rejected)
• The requirement to produce Orderly http://orderly-json.org/ and from this, JSONSchema http://json-schema.org/ for a particular service's JSON
• The requirement to produce RAML http://raml.org/ for a set of WS, and from this, WADL So for a particular WS, the following artifacts (files) would be made available to client developers:
• RAML
• WADL
• Orderly
• JSONSchema

Strictly not allowed: (examples from http://open.canada.ca/data/api/action/package_show?id=0016fe18-9539-42c9-8227-4bc1956c2a3e ):

• property names that do not break the spirit of proper structures. Example: "title_fra": "CanMatrix - géoréférencé - 063O04 - Wimapedi Lake, Manitoba", should be: "title": { "fr": "CanMatrix - géoréférencé - 063O04 - Wimapedi Lake, Manitoba" }
• Structure (map) embedded in values. Example: "presentation_form": "Map Digital | Carte numérique", should be: "presentation_form": { "en": "Map Digital", "fr": "Carte numérique" }
• Related to previous: XML or JSON embedded in JSON values. Example: spatial": "{\"type\": \"Polygon\", \"coordinates\": [[[-100.0, 55.25], [-100.0, 55.0], [-99.5, 55.0], [-99.5, 55.25], [-100.0, 55.25]]]}",
• Related to previous: Embedded list structure in JSON. Example: "keywords": "buildings,cartography,communications,earth sciences,geographic data,geographic information systems,geographical maps,geography,geomatics,lakes,land cover,maps,rivers - streams,terrain elevation,topographical maps,topography,toponymy,transportation" should be: "keywords": ["buildings", "cartography", "communications", "earth sciences", "geographic data", "geographic information systems", "geographical maps", "geography", "geomatics", "lakes", "land cover", "maps", "rivers - streams", "terrain elevation", "topographical maps", "topography", "toponymy", "transportation]",
• null values. Do not send. Example: "hash": "", "last_modified": null Note that as the semantics of having a value that is "" is unclear, it is not allowed. "" is of course != null, but is effectively for these purposes.

Reject metadata bloat that 1) should be pointed to in single URL in JSON; and 2) is not needed by developers. Examples (arguably), from above URL: result.private: false result.maintainer: null, result.relationships_as_object: [ ], result.data_series_issue_identification_fra: "063O04", result.catalog_type: "Geo Data | Géo", result.data_series_issue_identification: "063O04", result.ready_to_publish: true, result.maintenance_and_update_frequency: "As Needed | Au besoin", result.author_email: "open-ouvert@tbs-sct.gc.ca", result.state: "active", results.type: "dataset", resources. "cache_last_updated": null, "webstore_last_updated": null, "tracking_summary": ... "cache_url": null, "webstore_url": null, "position": 0,

Generally:

• we want to send minimal but complete metadata, and data
• we want the barrier to access to be minimal With the above rejected metadata, perhaps it could all be replaced with something like: "extendedMetadataUrl": "http://open.canada.ca/data/api/action/metadata/0016fe18-9539-42c9-8227-4bc1956c2a3e"

What we examined and rejected:

• See http://stackoverflow.com/a/14538774

We would very much like to work with the contributors to the WET draft, and perhaps discuss if some of the efforts here at AAFC might be merged in, or at the very least discussed.
I am going to fork the WET draft and put merge in what we are doing, and then see if this seems a sensible way forward to the group.

Looking forward to working with you! :-)

-Glen

[ydagenais]

Hi Glen,

Not sure if others have commented offline to you about this post but it's great! Thanks for sharing.

You mention that you both consume and produce internal and external web services. Are any of your web services writable? I wasn't sure if your comment about consuming was limited to querying 3rd party web services. If you have writable APIs, any lessons learned regarding authentication which related to the API spec that you could share? (not the backend magic/technology).

I think metadata bloat is another interesting aspect that deserves discussion. However, I don't know if it could ever be part of the spec; perhaps more of a best practice. Either way, it's an interesting topic. Another one would be the inclusion or exclusion of null values. e.g: http://stackoverflow.com/questions/11003424/should-json-include-null-values Not trying to upset metadata and json purists... :innocent:

[chrismajewski]

@gnewton Any more on this?