Department of Home Affairs REST API Standards - points of divergence

[TimGoodwill]

Whilst the baseline Victorian standard is very comprehensive, and we have borrowed from it quite a bit, the Department of Home Affairs’ REST API standard differs from that of the Victorian standard in a number of key areas where the standard is silent, or problematic for federal government and Home Affairs portfolio APIs.

Major points

• The Victorian standard is not interoperable with the Australian Consumer Data Standards – and in fact about as structurally different as possible. There is a risk that we are creating technical debt. The Australian Consumer Data Standards, developed to implement Consumer Data Right legislation, does not cover the government sector at this time, however it could conceivably do so in the future. Is there a clear justification for diverging from these existing, legislated national standards?
• Consistent top-level members: We will require an enterprise resource API entity body to provide a ‘data’ top-level member, and will support a ‘meta’ top level element, as per the Australian Consumer Data standards, UK Open Banking specifications and the JSON-API specification. The ‘meta’ top-level member is in use by portfolio and numerous other federal agencies. Note that we differentiate between business-resource APIs and UI/experience APIs, and do not impose the constraint on UI/experience APIs.
• There is little serious attention to API security:

1. Among the examples provided in the Victorian standard are potentially Personally Identifiable Information presented in the URL as resource identifiers. We specifically warn against this.
2. The statement in the Vic standards security section “API keys MUST be used for client authentication” is misleading – API keys are client identification, and are not secure or robust enough a mechanism to be considered ‘authentication’. This should be called out.
3. Mandating SHA256 is not a good idea. The department mandates that external facing certificates MUST be SHA512 - and the requirement will evolve. SHA-512 is actually faster on 64-bit processors.
4. There is no detail around JOSE implementation in the Victorian doc – in particular, the JWS Detached Content approach to document signing is an important feature of our implementation – base-JWT based JWS is not suitable for signed documents as the content is often too large for a http header.
5. There is an absence of a policy for providing protective marking for data. We are re-purposing the PSPF Email Protective Marking Standard message header ‘X-Protective-Marking’ for classified data to make use of the well-defined, inter-operable and versioned semantics.

Minor points with interoperability ramifications

• We use link notation consistent with the Consumer Data Standards and JSON-API for pagination and for HATEOAS, as it is much more succinct and readable than Victoria’s Link Description Object (LDO) schema. Link notation is not consistent within the Victorian standards doc.
• We use JSON-API style pagination – it is more flexible, with a common syntax for offset, page and cursor based pagination
• We use the ‘fields’ query parameter keyword, rather than ‘attributes’. Sparse fieldsets (‘fields’) is a flexible and well understood mechanism (Google, JSON-API).
• We use the ‘filter’ keyword rather than ‘filters’. ‘filter’ is pretty much an industry standard (Microsoft, Google). We advocate for the use of simple filtering, rather than creating an expectation that Microservices should implement non-trivial filtering functionality. It should be implemented only where tooling and use-cases warrant it.
• We use JSON-API style sorting notation – it is much simpler and cleaner.

Other Feedback - language issues and unhelpful constraints

• Specifying OpenAPI version 2.0 is unhelpful. We require developers to produce an OpenAPI version 3.0 spec wherever possible, but will support v2.0 when backend technology constrains choice. V3.0 was introduced in July 2017.

• Preferencing snake_case over camelCase is an odd choice, and unnecessary. Preference will largely come down to the technology stack – CamelCase for Java/JavaScript and snake_case for Python/Ruby. In terms of common usage, the weight is with camelCase at this point in time. We use camelCase rather than snake_case for member/field names and query parameter names. As per the JSON-API, restfulapi.net, google and Microsoft guidance and JavaScript standard convention (https://www.w3schools.com/js/js_conventions.asp).

• Our department’s API strategy is heavily predicated on Domain Driven Design, which is not mentioned in the Victorian standards document. Statements in the Victorian standards doc such as ‘APIs should be customer centric’ (‘API as a Product’ section), in the absence of an overarching Domain Driven Design context could definitely lead in the wrong direction. We have nuanced this statement in our policy.

• The abstraction matrix (abstraction section) is unnecessary. The ‘Basic’ level of abstraction would not be acceptable in a federal government department. Statement that has been applied to ‘Intermediate’ levels of abstraction, “NEVER directly expose the internal database table structure as is in your API” and “avoid exposing and sharing internal system identifiers”, and statements applied to the ‘advanced’ level of abstraction, namely the use of an API gateway and HATEOAS, should ideally apply to all resource APIs published within a federal department.

• The Victorian API lifecycle attempts to be a little more all-encompassing than is necessary or helpful. We have:

  1. Alpha – it’s in design / test in lower environments only and there might be a mock to play with but don’t expect any stability – good time to share your views.
  2. Beta – it’s in limited production release because it’s passed internal governance controls. But it’s hot off the press and early use might indicate the need for change. Build to it with risk that you might need to change.
  3. Published – it’s thoroughly tested and ready for prime time. You can build to it safely and if there are breaking changes then new separate versions will be released giving you plenty of time to upgrade.
  4. Deprecated – there’s a better version out that you should build to if you are starting fresh. But if you are still using this old one, we will support you until it’s retired.
  5. Retired – this API has been shut down. If you haven’t moved to a more recent version then your application is going to stop working.

• Our error structure is similar, but ‘extended’ to include additional data such as ‘helpUrl’ (more information on the error) and ‘location’ (part request part that predicated the error). A base schema across fed gov would be useful, however allowing extensions to the schema allows a degree of tailoring to the needs of a specific environment.

• Rate-limiting headers are API Management platform specific, and should not be prescribed. Ours follow the format ‘x-ratelimit-reset’ rather than ‘X-Rate-Limit-Reset’ as per the Victorian document.

Please consider.

[jordanwalsh23]

Thanks @TimGoodwill for the comprehensive feedback. I will attempt to break this down into a number of smaller issues that the working group can consider for incorporation into the standard.

The next working group meeting is in January, so I will aim to have a few proposed changes in PR form prior to then.

[TimGoodwill]

Thanks for your response. I am happy for you to close this issue and manage the feedback as individual issues, or in the context of the Working Group - whichever suits.