XML File Format/Open API Spec

[wilsonge]

So one of the things that now exists is a JSON/YAML format for webservices in an open source group based on the old swagger specification https://openapis.org/

I'm pretty sure we should use this instead of our current XML files (with relevent custom properties as required as on it's own it's not enough :) )

[chrisdavenport]

There are two aspects to this and I'm not sure there is an ideal solution. The first is the internal configuration stuff (like how a resource maps to the integration layer and how we transform internal variable names and types to external names/types and vice versa). The second is how we want to expose the resource profiles to public API consumers. The current XML files effectively provide both services but are geared primarily to the first since they don't conform to any accepted public standard.

So, my thinking was that we use some internal format, whatever that is (currently it's our custom XML, but it doesn't have to be), then run it through what should be a fairly simple process to generate one or indeed several different public formats. Swagger could be one of those, as could JSON-LD, ALPS and probably others. They all have their pros and cons. These internal files can also be used to automatically generate the routes.json file.

The internal files don't need to be web-accessible, which is good because we're currently exposing a lot of detail about internal configurations that we probably shouldn't. It's not a huge security issue, but if we can avoid exposing that stuff then fewer people are likely to moan at us for revealing too much information. ;-)

I'm not a great fan of using JSON for files that could need to be edited by humans at some point, especially when those files can get quite large. I don't have an opinion on YAML because I haven't had a reason to use it yet. Despite its heavyweight nature, XML is easy to edit and maintain by ordinary humans and is well-supported by tools. And I was planning on writing a DTD file so that editors can validate it on-the-fly.

It's good that the (new) Swagger specification allows the addition of custom metadata, but I think their intention is for that metadata to be for public consumption. Our configuration metadata is not only not intended for public use, but I would argue, would be unnecessary and confusing clutter.

So, yes let's support the (new) Swagger, but let's not use it for our internal files. What do you think?

[wilsonge]

XML files are no more private than yaml/json - so it terms of allowing access to our data it doesn't really matter what we go with - we need to be incredibly careful...

FWIW there's no reason why the routes.json file can't be (a part of) an Open API json/yaml file (see my JAB 2016 talk for more information which will be doing EXACTLY that :P)

In terms of Open API - I've been using it at work exactly because yaml is human friendly like XML, BUT I can shove it into swagger editor and http://editor.swagger.io/#/ edit it in front of my boss so he can get a feel for end users looking at docs etc. but YAML is much more user friendly to edit than JSON.

But there's a huge overlap between our new routing files and the xml files anyhow - not sure what the intended end goal is...

[chrisdavenport]

XML files are no more private than yaml/json - so it terms of allowing access to our data it doesn't really matter what we go with - we need to be incredibly careful...

That's true, but the point I was trying to make is that we should separate the internal configurational data from the external public-facing profile. The internal files can be placed somewhere that is not web-accessible; outside the web root, or in a directory with a Deny All htaccess rule on it.

FWIW there's no reason why the routes.json file can't be (a part of) an Open API json/yaml file (see my JAB 2016 talk for more information which will be doing EXACTLY that :P)

Bear in mind that all the routes need to be loaded on every single request, but only one profile needs to be loaded. We need to be mindful of performance here.

In terms of Open API - I've been using it at work exactly because yaml is human friendly like XML, BUT I can shove it into swagger editor and http://editor.swagger.io/#/ edit it in front of my boss so he can get a feel for end users looking at docs etc. but YAML is much more user friendly to edit than JSON.

I'm going to have to read up on YAML. It's not something I've had any reason to use before.

But there's a huge overlap between our new routing files and the xml files anyhow - not sure what the intended end goal is...

Exactly. And I want to try to re-organise the information in the current XML files a lot.

So, here's what I have in mind at the moment (and I'm open to argument on all aspects of it). There are three categories of configuration information that we need to support. There's the routes, other internal stuff and the public profiles. These are the constraints:

• We need to load all the routes on every request.
• We only need to load one internal profile per request. Or two for a linked request.
• There could easily be hundreds of profiles on some sites.
• The public profiles don't need to be used by our software, although they could be, as long as they can be generated by our software.
• There are multiple competing public profile formats. We only really need to pick one, but our track record on picking winners is not great so having the flexibility to generate any current or future format is useful.
• The internal stuff (including the routes) needs to be understandable and editable by humans.
• The administrator component is an alternative, user-friendly way of editing the internal stuff (including the routes).

Within that framework I'm not that fussed about exactly what formats we use or where the files are located and so on, although I would argue that JSON is not really for humans.

One possibility might be to use OpenAPI/Swagger internally but strip the internal configuration stuff when rendering it to the public.